Maintaining a fleet of thousands of Red Hat OpenShift clusters, as is typical in network far edge deployments, requires a disciplined and consistent approach to managing configuration. In an ideal world all of the clusters would be running the same versions of software from the OpenShift Container Platform through application workload. In reality, however, the fleet of clusters will contain a mix of software versions due to phased rollouts, regional differences, or other operational requirements that may pin a set of clusters to an older release.
Addressing the complexity of multiple versions can be achieved by building on top of a couple of very powerful tools. Red Hat’s GitOps (Argo CD) operator allows for single-source of truth definitions of clusters and their configuration and Red Hat Advanced Cluster Management’s Policy/GRC functionality allows the user to define a configuration once and apply it to large sets of clusters.
With the fleet of clusters running different Red Hat OpenShift versions, there are also multiple versions of configuration which need to be defined and managed. In the Red Hat far edge use cases, the customized platform configuration for specific use cases, e.g., 5G RAN Distributed Units (DU) is captured in Git. The automated tooling combines this configuration with a versioned reference configuration to build the Policy CRs that are applied to clusters by ACM. With each new release comes an updated baseline reference configuration which is then adapted to the production network and wrapped in Policies.
In prior OpenShift releases (4.13 and before), the GitOps ZTP infrastructure provided a single version of the reference configuration as the baseline for the generated configuration of all managed clusters via a ZTP plugin container. This reference configuration comes in two pieces. First, a single version-specific set of install-time manifests (CRs) to apply during DU cluster installation and second, a version-specific set of configuration CRs which provide cluster tuning, networking, storage configuration, etc. The install manifests are referenced by the user’s SiteConfig CRs and included in the generated installation CRs, and the configuration CRs are referenced by the user’s PolicyGenTemplate CRs.
Both the SiteConfig and the PolicyGenTemplate are CRs, which the user defines in Git. This single fixed version of the reference configuration made ongoing management of older clusters difficult as the new deployments moved to newer releases.
Requirements and solution
The complexities of simultaneously managing multiple OpenShift versions within a fleet of clusters at large scale places additional requirements on the management solution, including:
- Clusters must be able to easily move through the upgrade process
- Ongoing management of clusters on older versions must be maintained
In the 4.14 release, we have extended the ZTP plugin to allow users to add the versioned reference configuration to the Git repository alongside their customized configuration. The reference configuration can be easily extracted from the version specific ZTP plugin container and directly placed into Git. In order to allow users to associate a specific versioned reference configuration with their clusters and policies, both the SiteConfig and PolicyGenTemplate CRs have been extended to provide an explicit linkage to the desired copy in Git. In the SiteConfig CR users can select the correct version of install manifests through the searchPaths parameter. This cluster level parameter accepts multiple entries supporting both reference configuration and additional user provided installation manifests. These resources are combined to form the basis of site specific configuration passed to the installer.
In a similar way the PolicyGenTemplate CR has been extended to provide an explicit linkage to the desired reference configuration in Git. In the PolicyGenTemplate the user can select the correct source CRS (reference configuration) from which their custom content is added. In addition to the reference configuration extracted from the ZTP plug-in container, users can add their own content which will be bound into policies and applied to the managed clusters. As a result, users can rely on their Git repository as the single-source-of-truth by including specific reference configuration versions along with their adaptations and policy specifications.
How to use it
To deploy and manage multiple OpenShift versions simultaneously, users must extract version specific reference configuration from ztp-site-generate containers and prepare their Git repository. We will walk through this preparation for both parts of the reference configuration: the installation portion, used by SiteConfig, and the configuration portion used by PolicyGenTemplate (PGT).
The user needs to keep a copy of the reference configuration “extra manifests” for each deployed OpenShift version that will be used during cluster installation. To deploy multiple versions, a user must create a version specific folder to hold the extracted content. We recommend users to keep reference CRs and custom CRs in separate directories. For simultaneously deploying multiple versions, as a best practice, we suggest organizing the siteconfig directory grouped by cluster versions and to keep reference configurations at the same directory level of the cluster definition. The example below shows our recommendation on organizing version specific directory structure for allowing a multiple version deployment:
siteconfig
├── kustomization.yaml
├── version_4.13
│ ├── sno-1.yaml
│ ├── kustomization.yaml
│ ├── extra-manifest/ <-- reference manifests extracted from ZTP container
│ └── custom-manifest/ <-- user's custom manifests
└── version_4.14
├── sno-2.yaml
├── kustomization.yaml
├── extra-manifest/
└── custom-manifest/
The ZTP plugin can build version specific install configuration from the directory locations provided in the searchPaths parameter of the SiteConfig CR and pass it to the installer for deployment as shown below:
---
apiVersion: ran.openshift.io/v1
kind: SiteConfig
metadata:
name: "sno-1"
Spec:
clusters:
- clusterName: "example-sno"
extraManifests:
searchPaths:
- extra-manifest/
- custom-manifest/
Managing multiple versions of configuration in Git follows a similar pattern. To apply advanced configuration in a managed cluster by wrapping around policies, users need to keep reference source CRs alongside with policy definitions at the same directory level. Similar to SiteConfig, policy definitions can be grouped by cluster version as shown below:
policygentemplates
│ ├── kustomization.yaml
│ ├── version_4.13
│ │ ├── common-pgt.yaml
│ │ ├── group-pgt.yaml
│ │ ├── site-pgt.yaml
│ │ ├── kustomization.yaml
│ │ ├── namespaces.yaml
│ │ └── source-crs/
| | └── reference-crs/ <-- reference policy CRs extracted from ZTP container
| | └── custom-crs/ <-- user’s custom CRs
│ └── version_4.14
│ ├── common-pgt.yaml
│ ├── group-pgt.yaml
│ ├── site-pgt.yaml
│ ├── kustomization.yaml
│ ├── namespaces.yaml
│ └── source-crs/
| └── reference-crs/ <-- reference policy CRs extracted from ZTP container
| └── custom-crs/ <-- user’s custom CRs
Apart from allowing version independence, our recommended directory structure also provides several benefits to move through z-stream and y-stream upgrade processes. Those are as follows:
- z-stream version updates: Users can directly copy new patches and/or bug fixes included in the updated z-stream version to the reference configuration in the Git repository to provide the fixes.
- y-stream version upgrades: Users can simply create a new directory and populate version specific content as shown in the above example.
Therefore, the ongoing management of clusters on older versions can be maintained easily and any new configuration changes are only bound to the clusters which are upgraded to a newer version. In both cases, the GitOps operator will generate the needed Policy and ACM will apply the new changes to the specific managed clusters.
Installation and ongoing configuration management of OpenShift clusters is made significantly easier with GitOps ZTP-driven solutions. OpenShift 4.14 introduces features supporting simultaneous installation and management of heterogeneous versions within a single Git hierarchy. These features reduce the friction, risk, and challenge of adopting new z-stream and y-stream releases resulting in improved maintainability and directly reducing the risk of exposure to security vulnerabilities.
