Automate VM golden image management with OpenShift

June 3, 2025
Ben Oukhanov
Related topics:
CI/CDDevOpsGitOpsKubernetesVirtualization
Related products:
Red Hat OpenShift GitOpsRed Hat OpenShiftRed Hat OpenShift Virtualization

Creating and distributing golden images of virtual machines across Red Hat OpenShift clusters is crucial for platform engineers aiming to standardize environments and optimize operational efficiency. This article demonstrates how to automate this process using Red Hat OpenShift Virtualization, Red Hat OpenShift Pipelines, and Red Hat OpenShift GitOps.

OpenShift Pipelines provides Kubernetes-native CI/CD capabilities. Pipelines, Tasks, and PipelineRuns are managed as Kubernetes custom resources, enabling fully declarative workflows. OpenShift Virtualization extends these concepts to virtual machines (VMs), treating them as first-class Kubernetes objects.

By combining OpenShift Pipelines and OpenShift Virtualization, you can:

  • Automate the build of disk images.
  • Manage them in Git for versioned, auditable GitOps workflows.
  • Distribute them seamlessly across clusters with OpenShift GitOps.

This article builds upon the workflow described in Building VM Images Using Tekton and Secrets, enhancing it by introducing automated upload of disk images to a container registry and automating their import into different OpenShift clusters. The YAML manifests used in this article available in the kubevirt-golden-images GitHub repository.

Prerequisites

In order to follow along with this guide, you will need:

  • Red Hat OpenShift 4.17 or newer
  • Red Hat OpenShift Virtualization 4.17 or newer
  • Red Hat OpenShift Pipelines
  • Red Hat OpenShift GitOps

Build and upload new custom golden image using OpenShift Pipelines

This custom golden image pipeline imports a Red Hat Enterprise Linux (RHEL) image from the Red Hat registry, uses virt-customize to install Git package, creates a modified copy of the image, and then uploads it to the container registry. Figure 1 depicts this.

Custom Golden Image Pipeline
Figure 1: Custom golden image pipeline.

This process requires two secrets: one storing container registry credentials and another for Red Hat account credentials.

Container registry credentials

These credentials will be used by the disk-uploader tool when pushing containerDisk to the container registry that will contain the disk image:

oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: disk-uploader-credentials
type: Opaque
stringData:
  accessKeyId: "<ACCESS_KEY_ID>" # <QUAY_USERNAME>
  secretKey: "<SECRET_KEY>" # <QUAY_PASSWORD>
EOF

Note

If you are using Red Hat Quay, it is recommended to create a Robot Account that is associated with your Red Hat account with unique credentials and permissions. More information is available in the Quay documentation.

Workspace credentials

Workplace credentials store your Red Hat account password used to acquire Red Hat’s subscription to install packages:

oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: disk-virt-customize-workspace-credentials
type: Opaque
stringData:
  password: "<RH_ACCOUNT_PASSWORD>"
EOF

Example pipeline

Here is an example of the custom golden image pipeline that automates the workflow described in Figure 1:

oc apply -f - <<EOF
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: disk-uploader-pipeline
spec:
  workspaces:
    - name: data01
  params:
    - name: REDHAT_USERNAME
      description: "Red Hat username to be used in RHEL subscription"
      type: string
    - name: IMAGE_DESTINATION
      description: "Destination of the image in container registry"
      type: string
    - name: SECRET_NAME
      description: "Name of the secret which holds credential for container registry"
      type: string
  tasks:
    # Step 1: Imports base RHEL image into a PresistentVolumeClaim (PVC)
    - name: import-rhel-image
      taskRef:
        resolver: hub
        params:
          - name: catalog
            value: redhat-tekton-tasks
          - name: kind
            value: task
          - name: name
            value: modify-data-object
          - name: version
            value: ">=4.18.0"
      params:
        - name: manifest
          value: |-
            apiVersion: cdi.kubevirt.io/v1beta1
            kind: DataVolume
            metadata:
              generateName: rhel-9-5-guest-dv-
              annotations:
                cdi.kubevirt.io/storage.bind.immediate.requested: "true"
            spec:
              source:
                registry:
                  pullMethod: node
                  url: "docker://registry.redhat.io/rhel9/rhel-guest-image:9.5-1734523887"
              storage:
                volumeMode: Filesystem
                resources:
                  requests:
                    storage: 10Gi
        - name: waitForSuccess
          value: true
        - name: allowReplace
          value: true
        - name: setOwnerReference
          value: true
    # Step 2: Customizes image (e.g. install git, register with Red Hat)
    - name: disk-virt-customize
      taskRef:
        resolver: hub
        params:
          - name: catalog
            value: redhat-tekton-tasks
          - name: kind
            value: task
          - name: name
            value: disk-virt-customize
          - name: version
            value: ">=4.18.0"
      runAfter:
        - import-rhel-image
      workspaces:
        - name: data01
          workspace: data01
      params:
        - name: pvc
          value: "$(tasks.import-rhel-image.results.name)"
        - name: virtCommands
          value: |-
            sm-credentials $(params.REDHAT_USERNAME):file:/data01/password
            sm-register
            sm-attach auto
            install git
            sm-unregister
    # Step 3: Copies the customized disk to a new PresistentVolumeClaim (PVC)
    - name: copy-rhel-image
      taskRef:
        resolver: hub
        params:
          - name: catalog
            value: redhat-tekton-tasks
          - name: kind
            value: task
          - name: name
            value: modify-data-object
          - name: version
            value: ">=4.18.0"
      runAfter:
        - disk-virt-customize
      params:
        - name: manifest
          value: |-
            apiVersion: cdi.kubevirt.io/v1beta1
            kind: DataVolume
            metadata:
              generateName: rhel-9-5-copied-guest-dv-
              annotations:
                cdi.kubevirt.io/storage.bind.immediate.requested: "true"
            spec:
              source:
                pvc:
                  name: "$(tasks.import-rhel-image.results.name)"
                  namespace: "$(tasks.import-rhel-image.results.namespace)"
              storage: {}
        - name: waitForSuccess
          value: true
        - name: allowReplace
          value: true
        - name: setOwnerReference
          value: true
    # Step 4: Uploads the image to a container registry
    - name: disk-uploader
      taskRef:
        resolver: hub
        params:
          - name: catalog
            value: redhat-tekton-tasks
          - name: kind
            value: task
          - name: name
            value: disk-uploader
          - name: version
            value: ">=4.18.0"
      runAfter:
        - copy-rhel-image
      params:
        - name: EXPORT_SOURCE_KIND
          value: "pvc"
        - name: EXPORT_SOURCE_NAME
          value: "$(tasks.copy-rhel-image.results.name)"
        - name: VOLUME_NAME
          value: "$(tasks.copy-rhel-image.results.name)"
        - name: IMAGE_DESTINATION
          value: "$(params.IMAGE_DESTINATION)"
        - name: SECRET_NAME
          value: "$(params.SECRET_NAME)"
EOF

Next, run the example pipeline above:

oc create -f - <<EOF  
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: disk-uploader-pipeline-run-
spec:
  pipelineRef:
    name: disk-uploader-pipeline
  workspaces:
    - name: data01
      secret:
        secretName: disk-virt-customize-workspace-credentials
  params:
    - name: REDHAT_USERNAME
      value: <VALUE>
    - name: IMAGE_DESTINATION
      value: <VALUE> # e.g. quay.io/rhel9/rhel-guest-custom:9.5
    - name: SECRET_NAME
      value: disk-uploader-credentials
  # This resolves an error due to guestfish lacking permission access to the disk img file
  taskRunSpecs:
    - pipelineTaskName: disk-virt-customize
      podTemplate:
        securityContext:
          fsGroup: 107
          runAsUser: 107
EOF

Distribute the custom golden image using OpenShift GitOps

OpenShift GitOps (Argo CD) manages and updates the HyperConverged configuration across OpenShift clusters by syncing it from a centralized GitHub repository, as shown in Figure 2.

Distribute Custom Golden Image
Figure 2: Distribute custom golden image.

The HyperConverged custom resource (CR) manages golden images by deploying DataImportCron objects. This periodically pulls golden images from a container registry to the cluster, and golden images can be selected when provisioning new virtual machines from InstanceTypes.

Argo CD labeling

Add a new label to the openshift-cnv namespace, which will allow the Argo CD application to update an existing HyperConverged configuration:

oc label namespace openshift-cnv argocd.argoproj.io/managed-by=openshift-gitops

Argo CD application

The following is a new Argo CD application that will update an existing HyperConverged configuration in the internal and external cluster:

oc apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: argocd-sample
  namespace: openshift-gitops
spec:
  generators:
    - list:
        elements:
          - name: in-cluster
            namespace: openshift-cnv
            server: https://kubernetes.default.svc
          - name: external-cluster
            namespace: openshift-cnv
            server: https://1.2.3.4:6443 # Example external API server
  template:
    metadata:
      name: argocd-sample-{{name}}
    spec:
      project: default
      source:
        repoURL: https://github.com/codingben/kubevirt-golden-images
        targetRevision: HEAD
        path: argocd-manifests
      destination:
        server: "{{server}}"
        namespace: "{{namespace}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: false
EOF

Next, check the status of the Argo CD application:

oc get applications.argoproj.io -n openshift-gitops

Then ensure that the status is Synced:

NAME            SYNC STATUS   HEALTH STATUS
argocd-sample   Synced        Healthy

Verify the golden image's existence in the cluster by checking for a DataSource that references it:

oc get datasource rhel-guest-custom

Example golden image usage

Create a new virtual machine using a custom golden image (rhel-guest-custom) deployed to the cluster:

oc apply -f - <<EOF
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: rhel-9-beige
  namespace: sample-app
spec:
  dataVolumeTemplates:
    - metadata:
        name: rhel-9-beige
      spec:
        sourceRef:
          kind: DataSource
          name: rhel-guest-custom
          namespace: sample-app
        storage:
          resources: {}
  instancetype:
    name: u1.medium
  preference:
    name: rhel.9
  runStrategy: Always
  template:
    spec:
      domain:
        devices: {}
      volumes:
        - dataVolume:
            name: rhel-9-beige
          name: rootdisk
        - cloudInitNoCloud:
            userData: |
              #cloud-config
              chpasswd:
                expire: false
              password: idvv-ykvl-1x6j
              user: rhel
          name: cloudinitdisk
EOF

Conclusion

By combining OpenShift Pipelines, OpenShift Virtualization, and OpenShift GitOps, you can fully automate the lifecycle management of golden images across OpenShift clusters. This approach provides standardized and consistent VM environments, Git-backed auditable workflows, and seamless distribution of images across clusters.

Explore OpenShift Container Platform capabilities or activate a no-cost trial.