How to back up and restore virtual machines with OpenShift

If you are planning a move to Red Hat OpenShift Virtualization, you may have a list of features that need to be checked to demonstrate similar capabilities to more established virtualization platforms. This article will guide you through one such feature: backing up and restoring virtual machines. There are several partner products that can simplify this operation, but these steps show a quick way to get started directly with the OpenShift APIs for Data Protection, provided by Red Hat.

Preparation

These instructions assume you have administrative access to an OpenShift cluster. On this cluster, install and configure the following three Operators:

• Red Hat OpenShift Virtualization: Refer to these instructions to install OpenShift Virtualization and create a HyperConverged Operator instance.
• OpenShift APIs for Data Protection (OADP): Refer to these instructions to install OADP.
• MinIO: These steps assume a local installation of MinIO, for non-production backup and restore on the running cluster. Refer to these instructions to install the MinIO operator and configure a tenant and a bucket from the command line or OpenShift console, or read the next section to use the MinIO operator management interface. If the MinIO operator is not available to you, you can also use the “Install MinIO Server” section of these instructions as an example of how to set up standalone MinIO storage. In any case, make sure to record the access key ID and the secret key. For this example, the tenant name is "minio", the namespace is "oadp-minio-tenant", and the new bucket's name is "oadp-bucket".

MinIO Console Configuration (optional)

The MinIO operator comes with a convenient management console that can be used to provision and configure storage, and to create and browse buckets. If you would like to try it out, first find the MinIO console pod and forward the port with oc:

$ oc get pod -n minio-operator -l app=console -o jsonpath='{.items..metadata.name}'
console-f5db76d94-5qtfc
$ oc port-forward -n minio-operator console-f5db76d94-5qtfc 9443:9443
Forwarding from  127.0.0.1:9443 -> 9443
Forwarding from  [::1]:9443 -> 9443

Log in to https://localhost:9443 with a browser, and you should be asked for a login token. Get it like this, from another terminal:

$ oc get secret -n minio-operator console-sa-secret -o jsonpath={.data.token} |  base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6IjB2RVRlSVdFdnNWdGtoS3pQSHZESm5BSXdtQlg3WER0MzE4………

It is a very long string, paste it directly into the “JWT” box. Before creating the tenant, create a namespace to hold it, and retrieve the user ID from its allowable supplemental groups:

$ oc new-project oadp-minio-tenant
Now using project "oadp-minio-tenant" on server
"https://api.ocp.captive.lab:6443".
$ oc get namespace oadp-minio-tenant -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}' | cut -d / -f 1
1000760000

Now you can create a tenant from the MinIO console. Make sure to enable the “Security Context” option on the “Configure” section, and copy this ID to the “Run As User”, “Run As Group”, and “FsGroup” fields. For this example, the number of servers was changed down to 1 because it is running on a single-node OpenShift cluster. The tenant may not start with the default of 4 because its pods cannot be independently scheduled on a single node. Tenant creation may take a few minutes, but if it appears to be stuck on "Provisioning initial users", you may need to restart the MinIO operator pod:

 $ oc delete pod -n minio-operator -l name=minio-operator
pod "minio-operator-bb56876d7-mjzsr" deleted

Preparing the virtual machine

See this blog post to create a single virtual machine (VM), if you haven’t already. To make sure that OADP is correctly able to back up and restore the VM, it is a good idea to create some kind of test data to verify after the restore is complete. One quick way is to save some random numbers from inside the VM, like:

[fedora@testvm ~]$ dd if=/dev/urandom of=verification.test bs=1M count=512 status=progress
349175808 bytes (349 MB, 333 MiB) copied, 1 s, 349 MB/s
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 1.53152 s, 351 MB/s
[fedora@testvm ~]$ sha1sum verification.test
481c9d852b3679a5525d542ec0ce4784ab4bf188 verification.test

After the restore, the VM should boot up and show the same file with the same data. The rest of this post will assume that the VM is named testvm, in an oadp-test namespace.

Configuring OADP for MinIO

Now it is time to create a DataProtectionApplication (DPA), which will point OADP to the MinIO storage previously configured. Creating a good DataProtectionApplication should automatically result in the creation of a BackupStorageLocation.

Providing MinIO Secrets to OADP

The first step is to tell OADP how to log in to MinIO. Create a file to hold the access credentials for the MinIO bucket, filling in the key fields as appropriate. If you downloaded credentials.json from the MinIO console, they will be stored in the accessKey and secretKey fields:

$ cat <<EOF > minio-credentials
[minio-profile]
aws_access_key_id=<insert access key here>
aws_secret_access_key=<insert secret key here>
EOF

Use this file to create a secret in the openshift-adp namespace, like this:

$ oc create secret generic minio-credentials -n openshift-adp --from-file cloud=minio-credentials

Creating a DataProtectionApplication

Now create a DataProtectionApplication, with fields set to values that match values from the cluster:

$ cat <<EOF > dpa.yaml
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa
  namespace: openshift-adp
spec:
  backupLocations:
    - velero:
        provider: aws
        config:
          profile: minio-profile
          s3ForcePathStyle: 'true'
          s3Url: 'https://minio.oadp-minio-tenant.svc.cluster.local'
          region: anywhere
          insecureSkipTLSVerify: 'true'
        credential:
          key: cloud
          name: minio-credentials
        default: true
        objectStorage:
          bucket: oadp-bucket
          prefix: oadp-vm-test
  configuration:
    velero:
      defaultPlugins:
        - csi
        - openshift
        - aws
        - kubevirt
EOF
$ oc create -f dpa.yaml

Some things to notice:

• The provider field is still set to aws even though there is no real connection to any Amazon service— MinIO is API-compatible with S3, so OADP uses the same aws plugin to access it.
• The use of the kubevirt plugin in the spec.configuration.velero.defaultPlugins list—this is needed for freezing the guest filesystem to ensure data consistency while taking a snapshot.
• The use of the csi plugin in the plugins list,  which is required for backing up virtual machines.
• The minio-profile profile—this should match the heading from minio-credentials file above.
• The insecureSkipTLSVerify option, with an HTTPS URL—the MinIO operator sets up HTTPS by default, and the easiest way to test is to skip TLS verification. If you want to do this the right way, set a caCert field under the objectStorage section and include the base64-encoded certificate.
• The URL used for MinIO contains the tenant name, so a different URL will need to be provided to the DataProtectionApplication depending on the name chosen for the tenant.

Check the status of the DataProtectionApplication to make sure it is reconciling correctly:

$ oc get dpa dpa -n openshift-adp -o jsonpath='{.status}'
{"conditions":[{"lastTransitionTime":"2024-05-21T16:52:35Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

Also check the status of the BackupStorageLocation. Note that the name of the BSL is the name of the DPA with a -1 appended:

$ oc get bsl -n openshift-adp
NAME PHASE LAST VALIDATED AGE DEFAULT
dpa-1 Available 2s 22h true

The DPA must be marked reconciled and the BSL must be marked Available before proceeding with the next steps.

Backups and restores

When taking a backup, VM volumes are saved by taking snapshots via CSI. CSI backups do not save a copy of the VM data itself, but rather just the Kubernetes objects that can be used to re-establish access to the same volume when the VM is removed and restored. If you need to back up the actual VM data, it is necessary to configure a CSI backup with the Data Mover feature enabled, which will upload the VM data to the backup storage. These instructions will show a CSI-only backup, followed by a backup with Data Mover enabled.

CSI

The DPA should already be configured for CSI-only backups from the previous section, so you can proceed directly to creation of a Backup.

Create a backup

Initiate a backup by creating a CRD with the VM’s namespace:

$ cat <<EOF > backup.yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: testvm-backup
  namespace: openshift-adp
spec:
  includedNamespaces:
    - oadp-test
  storageLocation: dpa-1
EOF
$ oc create -f backup.yaml

Watch for the backup to move through its phases, collecting everything to put into backup storage:

$ oc get backup testvm-backup -n openshift-adp -o yaml -o jsonpath='{.status.phase}{"\n"}' -w
InProgress
InProgress
InProgress
Finalizing
Completed

Now you can remove the VM:

$ oc delete virtualmachine testvm -n oadp-test
virtualmachine.kubevirt.io "testvm" deleted

If you have brought up the MinIO console, you should be able to browse the bucket to see what items are contained in the backup. Notice that there are no PVs, or really any files big enough to represent a VM disk. The contents are still present in the backing storage, and there is a CSI snapshot with enough information to quickly attach the restored VM to those contents. In this case, the cluster is running LVM storage, and the VolumeSnapshot item shows the UUID of the snapshot in LVM:

$ oc get volumesnapshot velero-testvm-jdmsb -n oadp-test -o jsonpath='{.metadata.annotations.velero\.io/csi-volumesnapshot-handle}{"\n"}'
e58b8c90-77c1-4cd6-98d4-e47cc6fc1eb9

Logging in to the cluster shows that there is still a logical volume present for this snapshot even though the VM itself has been deleted:

$ ssh core@ocp
[core@ocp ~]$ sudo lvdisplay vg1/e58b8c90-77c1-4cd6-98d4-e47cc6fc1eb9
--- Logical volume ---
LV Path                /dev/vg1/e58b8c90-77c1-4cd6-98d4-e47cc6fc1eb9
LV Name                e58b8c90-77c1-4cd6-98d4-e47cc6fc1eb9
VG Name                vg1
LV UUID                dFGbkd-ovgT-3ctK-3i5S-d5JK-utv4-6lgJPN
LV Write Access        read only
LV Creation host, time ocp, 2024-07-07 00:40:03 +0000
LV Pool name           thin-pool-1
LV Status              available
# open                 0
LV Size                30.00 GiB
Mapped size            16.11%
Current LE             7680
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     8192
Block device           253:15

Create a restore

With the VM deleted, create a Restore pointing to the backup you just saved:

$ cat <<EOF > restore.yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: testvm-restore
  namespace: openshift-adp
spec:
  backupName: testvm-backup
  restorePVs: true
EOF
$ oc create -f restore.yaml

Note the use of the restorePVs flag! Watch the status like before:

$ oc get restore testvm-restore -n openshift-adp -o yaml -o jsonpath='{.status.phase}{"\n"}' -w
InProgress
InProgress
InProgress
Completed

Now log in to the VM, and you should see the same test data has been restored:

[fedora@testvm ~]$ cat sum.txt
e241d30591670ae0eb2ab58a216d7b24bef9f642 verification.test
[fedora@testvm ~]$ sha1sum verification.test
e241d30591670ae0eb2ab58a216d7b24bef9f642 verification.test

CSI with DataMover

Now to get the backup to include the actual VM disk data, edit or re-create the DataProtectionApplication and configure it to use Data Mover:

$ oc delete dpa -n openshift-adp dpa
dataprotectionapplication.oadp.openshift.io "dpa" deleted
$ cat <<EOF > dpa.yaml
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa
  namespace: openshift-adp
spec:
  backupLocations:
    - velero:
        provider: aws
        config:
          profile: minio-profile
          s3ForcePathStyle: 'true'
          s3Url: 'https://minio.oadp-minio-tenant.svc.cluster.local'
          region: anywhere
          insecureSkipTLSVerify: 'true'
        credential:
          key: cloud
          name: minio-credentials
        default: true
        objectStorage:
          bucket: oadp-bucket
          prefix: oadp-vm-test
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      defaultPlugins:
        - csi
        - openshift
        - aws
        - kubevirt
EOF
$ oc create -f dpa.yaml

The big difference here is the addition of the nodeAgent configuration. During backup and restore, this will create a pod with an attached PV that is a snapshot of the target volume, so that it can copy the VM data.

Create a backup

To try this out, create another backup, making sure to set the snapshotMoveData flag:

$ cat <<EOF > backup.yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: testvm-backup-datamover
  namespace: openshift-adp
spec:
  snapshotMoveData: true
  includedNamespaces:
    - oadp-test
  storageLocation: dpa-1
EOF
$ oc create -f backup.yaml

This will take a little longer than the CSI-only backup:

$ oc get backup testvm-backup-datamover -n openshift-adp -o yaml -o jsonpath='{.status.phase}{"\n"}' -w
InProgress
InProgress
InProgress
WaitingForPluginOperations
Finalizing
Completed

If you have brought up the MinIO console, check the bucket for a “kopia” directory, in the oadp-vm-test prefix. It should contain a bunch of larger files that look suspiciously like they could be used to restore a whole VM.

Once again, delete the VM and try to restore it.

$ oc delete virtualmachine testvm -n oadp-test
virtualmachine.kubevirt.io "testvm" deleted

Create a restore

The new restore is effectively the same as before, no special flags are needed to indicate a Data Mover restore:

$ cat <<EOF > restore.yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: testvm-restore-datamover
  namespace: openshift-adp
spec:
  backupName: testvm-backup-datamover
  restorePVs: true
EOF
$ oc create -f restore.yaml

Watch the progress:

$ oc get restore testvm-restore-datamover -n openshift-adp -o jsonpath='{.status.phase}{"\n"}' -w
InProgress
InProgress
InProgress
WaitingForPluginOperations
Completed

After a few minutes, the VM should start. Log in to the VM again and verify the data:

[fedora@testvm ~]$ cat sum.txt
e241d30591670ae0eb2ab58a216d7b24bef9f642  verification.test
[fedora@testvm ~]$ sha1sum verification.test
e241d30591670ae0eb2ab58a216d7b24bef9f642  verification.test

Conclusion

These steps have shown a fairly simplistic approach to testing out VM backup and restore on a non-production cluster configuration. After reading this post, hopefully you have:

• Confidence that OpenShift is able to provide backup and restore capabilities for your virtual machines.
• A clearer idea of what the “Data Mover” option provides for VM backup and restore.

OADP has many more options for storage and configuration, please feel free to consult the documentation and experiment with its capabilities.