Build on multi-arch clusters with builds for Red Hat OpenShift

In this article, we'll talk about the builds for Red Hat OpenShift operator and how it simplifies building multi-arch builds on clusters with compute nodes of mixed architecture. This lets you create a single build object as opposed to separate builds for each architecture, as needed for the standard build/BuildConfig-based method in Red Hat OpenShift. The operator also takes care of creating and pushing a manifest list of the resulting images.

Building a manifest-listed image

Let's walk through the process of building a manifest-listed image with 2 architectures on a multi-arch compute cluster.

We'll accomplish this in the following steps:

1. Prerequisites: A running multi-arch compute cluster and a client to interact with it. For this example, our cluster has nodes of amd64, and arm64.
2. Install and configure the operator.
3. Create a new project.
4. Configure and add the ClusterBuildStrategy to our cluster.
5. Create the build YAML and start the build.
6. Verify the images are built for all architectures.
7. Create and push a manifest list.

First, we need to install the oc client utility to interact with our multi-arch compute cluster. Check that oc is installed correctly using the oc version command. If you do not have oc, you can follow the installation docs.

$ ./oc version
Client Version: 4.19.4
Kustomize Version: v5.5.0
Server Version: 4.19.2
Kubernetes Version: v1.32.5

To access a multi-arch compute cluster with the builds for Red Hat OpenShift operator available, you'll need OpenShift Container Platform 4.17 or later. You will need your access credentials, such as the cluster URL and kubeconfig file provided during installation.

Our cluster looks like this:

$ oc get nodes --label-columns='kubernetes.io/arch'
NAME                                      STATUS   ROLES                  AGE    VERSION   ARCH
aarch64-01.example.com    Ready    worker                 426d   v1.32.5   arm64
aarch64-02.example.com    Ready    worker                 426d   v1.32.5   arm64
x86-04.example.com        Ready    worker                 417d   v1.32.5   amd64
x86-05.example.com        Ready    worker                 408d   v1.32.5   amd64
x86-01.example.com        Ready    control-plane,master   426d   v1.32.5   amd64
x86-02.example.com        Ready    control-plane,master   426d   v1.32.5   amd64
x86-03.example.com        Ready    control-plane,master   426d   v1.32.5   amd64

Follow the installation section of the builds for Red Hat OpenShift documentation to install the operator using the web console or via CLI.

Use the verification steps included in the docs to ensure everything installed properly:

$ oc get openshiftbuilds
NAME      AGE
cluster   23h
$ oc get shipwrightbuilds
NAME                    AGE
cluster-8hjrb           23h
openshift-builds        385d
openshift-builds-mrda   351d
$ oc get pods -n openshift-builds
NAME                                                  READY   STATUS    RESTARTS   AGE
openshift-builds-operator-b59cc7f4d-47nxd             2/2     Running   0          23h
shared-resource-csi-driver-node-2m6jk                 2/2     Running   0          23h
...
shared-resource-csi-driver-webhook-597d666db8-6nrpf   1/1     Running   0          23h
shipwright-build-controller-6b966c8dd9-45x57          1/1     Running   0          23h
shipwright-build-webhook-f656dd877-2skzb              1/1     Running   0          23h

Now that the operator is installed, you can use your cluster to create builds for multiple architectures simultaneously. We'll do this by using the multiarch-native-buildah ClusterBuildStrategy. This creates jobs that place a build on nodes of the architectures you choose in your configuration and then stitches them together into a manifest list/index image. 

You can find the YAML files used in this section here.

Let's use oc to log in to the cluster:

$ oc login --token=sha256~*********************************** --server=https://example.com:6443
Logged into "https://example.com:6443" as "kube:admin" using the token provided.
Using project "default".

Create a new project called demo:

$ oc new-project demo
Now using project "demo" on server "https://example.com:6443"

Next, create a role and role bindings to ensure that you have permissions to work in the cluster:

$ oc apply -f ./clusterrole_multiarch_native_buildah_cr.yaml.yaml
role.rbac.authorization.k8s.io/multiarch-native-buildah-pipeline created
$ oc apply -f ./rolebinding_multiarch_native_buildah_cr.yaml
rolebinding.rbac.authorization.k8s.io/multiarch-native-buildah-pipeline created
$ oc apply -f ./rolebinding_multiarch_native_buildah_scc_okd_cr.yaml
rolebinding.rbac.authorization.k8s.io/multiarch-native-buildah-pipeline-scc-privileged created

Confirm that they were created correctly:

$ oc get clusterrole | grep multiarch
multiarch-native-buildah-pipeline
$ oc get rolebinding | grep multiarch
multiarch-native-buildah-pipeline                  ClusterRole/multiarch-native-buildah-pipeline   2m44s
multiarch-native-buildah-pipeline-scc-privileged   ClusterRole/system:openshift:scc:privileged     2m35s

Apply the multiarch-native-buildah ClusterBuildStrategy to the cluster:

$ oc apply -f ./buildstrategy_multiarch_native_buildah_cr.yaml
clusterbuildstrategy.shipwright.io/multiarch-native-buildah created
$ oc get clusterbuildstrategies | grep multiarch
multiarch-native-buildah        35s

Now that the ClusterBuildStrategy exists in the cluster and has the permissions to run, you can use it to create your builds. Let's create a build YAML that specifies the 2 architectures in the cluster.

We're pulling from the example in the Shipwright documentation for the strategy.

$ cat multiarch-native-buildah-example.yaml
---
apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: multiarch-native-buildah-example
  namespace: demo
spec:
  source:
    type: Git
    git:
      url: https://github.com/shipwright-io/sample-go
    contextDir: docker-build
  strategy:
    name: multiarch-native-buildah
    kind: ClusterBuildStrategy
  paramValues:
    - name: architectures
      values:
        - value: "amd64"
        - value: "arm64" 
    - name: build-contexts
      values:
        - value: "ghcr.io/shipwright-io/shipwright-samples/golang:1.18=docker://ghcr.io/shipwright-io/shipwright-samples/golang:1.18"
    # The buildah `--from` replaces the first FROM statement
    - name: from
      value: "" # Using the build-contexts for this example
    # The runtime-stage-from implements the logic to replace the last stage FROM image of a Dockerfile
    - name: runtime-stage-from
      value: docker://gcr.io/distroless/static:nonroot
    - name: dockerfile
      value: Dockerfile
  output:
    image: image-registry.openshift-image-registry.svc:5000/build-examples/taxi-app

Now, you can trigger the build and see that it succeeds on all architectures:

$ oc create -f multiarch-native-buildah-example.yaml
build.shipwright.io/multiarch-native-buildah-example created
$ oc get BuildRun
NAME                                     SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
multiarch-native-buildah-example-52ff9   True        Succeeded   91s         24s

$ oc get Jobs
NAME                                                       STATUS     COMPLETIONS   DURATION   AGE
multiarch-native-buildah-example-52ff9-dr7dv-job-amd64     Complete   1/1           24s        72s
multiarch-native-buildah-example-52ff9-dr7dv-job-arm64     Complete   1/1           19s        72s

If you look at the logs for the build, you can see that artifacts were built for both architectures and the BuildRun completed successfully after creating a manifest list of the 2 built images.

$ oc logs multiarch-native-buildah-example-zlk56-7rqc4-pod step-package-manifest-list-and-push
...
[INFO] Creating manifest list
d96bcabffb37bc10d84399b9955713ee59d2894da7b6980e4b8c14b6230b9b2f
[INFO] Adding the amd64 manifest to the manifest list
d96bcabffb37bc10d84399b9955713ee59d2894da7b6980e4b8c14b6230b9b2f: sha256:21284365a4ee6daf0fa5270826bbb5248599ec9d4d46a4d000532b1cd7073ad8
[INFO] Adding the arm64 manifest to the manifest list
d96bcabffb37bc10d84399b9955713ee59d2894da7b6980e4b8c14b6230b9b2f: sha256:1fec58eb2d9ad6ddbd3ff49410c0a1e8441b80e680b550e35f254c40cb43fe8b
[INFO] Pushing the manifest list taxi-app to the registry as image-registry.openshift-image-registry.svc:5000/build-examples/taxi-app
Getting image list signatures
Copying 2 images generated from 2 images in list
Copying image sha256:21284365a4ee6daf0fa5270826bbb5248599ec9d4d46a4d000532b1cd7073ad8 (1/4)
Getting image source signatures
Copying blob sha256:07e19bf3ffe596fa6f41a082b58c7e6397a8aa24ba7d601824b148a47ddfcd46
[INFO] Manifest list pushed successfully. BuildRun succeeded.

Summary

You have successfully created and run a build of two different architectures using the multiarch-native-buildah ClusterBuildStrategy on our multi-arch cluster and built a manifest list of the resulting images, all in a single build object. Using builds for Red Hat OpenShift simplifies the process of using build and BuildConfig objects manually, as we detailed in our previous article in this series.