This article shows an example of using the Operator Lifecycle Manager (OLM) bundle deployment architecture to deploy a Red Hat OpenShift or other Kubernetes Operator. You will learn how to use OLM and the Operator SDK (both components of the Kubernetes Operator Framework) together to deploy an Operator.
About the example
I tested the Operator Lifecycle Manager example on both an OpenShift 4.6 cluster (via Red Hat CodeReady Containers) and a local Kubernetes Kind 1.18 cluster. I used Operator SDK 1.0 and OLM 0.15.0 for this example.
Scaffolding the Operator
A bundle is an Operator packaging construct that contains an Operator definition and manifests used to determine how the Operator is deployed onto a Kubernetes cluster. The original OLM package manifest format has migrated to the bundle format.
In this example, we will use Operator SDK 1.0 to generate an Operator bundle and create the Operator for deployment. Enter the following operator-sdk commands to begin scaffolding the sample Operator:
$ operator-sdk init --domain=example.com --repo=github.com/example-inc/doo-operator $ operator-sdk create api --group cache --version v1 --kind Doo --resource=true --controller=true
Working with container images in Operator SDK
When operator-sdk generates or scaffolds an Operator, it does not include any logic about your application. You will need to include information about how to install and manage your application. Like any other Operator, the resulting code is built into a container image. In this example, the Operator image is named quay.io/username/doo-operator:v0.0.1.
Entering the operator-sdk init command creates a project structure. This command also creates a Makefile that you can use for various build and deployment tasks. Enter the following to build an Operator image using a Makefile:
$ make docker-build docker-push IMG=quay.io/username/doo-operator:v0.0.1
Note: See the Operator SDK documentation for more about building Operators with operator-sdk and Golang.
Building an Operator bundle image
Apart from the Operator image, an operator bundle is an OLM-prescribed format for holding Operator metadata. The metadata contains everything Kubernetes needs to know to use the Operator, including its custom resource definitions (CRDs), required role-based access and control (RBAC) roles and bindings, dependency tree, and more.
Generate a bundle
You can use the operator-sdk-generated Makefile to create a bundle for your Operator:
$ make bundle IMG=quay.io/username/doo-operator:v0.0.1
This command generates an on-disk set of bundle manifests. Here is an example of the directory structure for a generated bundle:
bundle ├── manifests │ ├── cache.example.com_dooes.yaml │ ├── doo.clusterserviceversion.yaml │ └── doo-metrics-reader_rbac.authorization.k8s.io_v1beta1_clusterrole.yaml ├── metadata │ └── annotations.yaml └── tests └── scorecard └── config.yaml
The make bundle command also creates a Dockerfile (bundle.Dockerfile), which is used to build a bundle image. The bundle image is an Open Container Initiative (OCI)-compliant image that holds the generated on-disk bundle manifest and metadata files.
Create and push the bundle image
In this example, we'll name the Operator bundle image as follows:
quay.io/username/doo-operator-bundle:v0.0.1
To create and push the image, run the following Makefile targets:
$ make bundle-build BUNDLE_IMG=quay.io/username/doo-operator-bundle:v0.0.1 $ make docker-push IMG=quay.io/username/doo-operator-bundle:v0.0.1
Building the Operator index image
Another OLM concept is the index image. This container image serves an application programming interface (API), which describes information about your sample Operator. The index image includes information from your bundle image by running opm, an Operator registry command:
$ opm index add --bundles quay.io/username/doo-operator-bundle:v0.0.1 --tag quay.io/username/doo-operator-index:v0.0.1 $ podman push quay.io/username/doo-operator-index:v0.0.1
The index image holds an SQLite database with bundle definitions. It also runs a gRPC service when the image is executed. The gRPC service lets consumers query the SQLite database about the Operators the index contains.
Note: You can download the opm command from the Operator Framework's Operator registry.
Running the bundle index image
At this point, you should have OLM installed on your Kubernetes cluster. OLM is installed by default on OpenShift clusters. You can use Operator SDK's olm install command to install OLM on any other Kubernetes cluster manually.
Once you have an index image, deploy it by creating an OLM CatalogSource resource. For our sample Operator, we create the CatalogSource as follows:
$ cat <<EOF | kubectl create -f - kind: CatalogSource metadata: name: doo-operator namespace: operators spec: sourceType: grpc image: quay.io/username/doo-operator-index:v0.0.1 EOF
This CatalogSource is created in an existing namespace, created by OLM, named operators. On OpenShift clusters, this namespace is called openshift-operators. When you create the CatalogSource, it causes the index image to be executed as a pod. You can view it as follows:
$ kubectl -n operators get pod --selector=olm.catalogSource=doo-operator NAME READY STATUS RESTARTS AGE doo-operator-79x8z 1/1 Running 0 136m
You can look at the pod's log to ensure the image is serving the gRPC API:
$ kubectl -n operators logs pod/doo-operator-79x8z time="2020-10-05T13:17:04Z" level=info msg="Keeping server open for infinite seconds" database=/database/index.db port=50051 time="2020-10-05T13:17:04Z" level=info msg="serving registry" database=/database/index.db port=50051
Deploying the Operator
We use an OLM subscription resource to trigger a specific Operator deployment. With the following command, we create a Subscription that triggers the deployment of our sample Operator:
$ cat <<EOF | kubectl create -f - apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: doo-subscription namespace: operators spec: channel: alpha name: doo source: doo-operator sourceNamespace: operators EOF
Notice that the Subscription is created in the pre-existing operators namespace so that OLM will create the sample Operator in that same namespace. (Note, again, that on OpenShift clusters, the namespace is called openshift-operators.)
Verifying the Operator
We can use the following commands to verify the sample Operator is running. Let's start by verifying the Subscription has been created:
$ kubectl -n operators get subscription NAME PACKAGE SOURCE CHANNEL doo-subscription doo doo-operator alpha
Next, verify the operator CSV has successfully deployed:
$ kubectl -n operators get csv NAME DISPLAY VERSION REPLACES PHASE doo.v0.0.1 doo-operator 0.0.1 Succeeded
Finally, verify the Operator is running:
$ kubectl -n operators get pod NAME READY STATUS RESTARTS AGE doo-controller-manager-6c4bdf7db6-jcvpn 2/2 Running 0 10m
Testing the Operator
We can test the sample Operator by creating a CustomResource that the sample Operator is watching.
Create the CustomResource in the default namespace as follows:
$ cat <<EOF | kubectl -n default create -f -
{
"apiVersion": "cache.example.com/v1",
"kind": "Doo",
"metadata": {
"name": "doo-sample"
},
"spec": {
"foo": "bar"
}
}
EOF
Your sample Operator should respond to the creation of the CustomResource by checking the sample Operator log:
$ kubectl -n operators logs pod/doo-controller-manager-6c4bdf7db6-jcvpn -c manager
2020-10-05T13:29:52.175Z DEBUG controller Successfully Reconciled{"reconcilerGroup": "cache.example.com", "reconcilerKind": "Doo", "controller": "doo", "name": "doo-sample", "namespace": "default"}
Create a unique namespace and OperatorGroup
Examples so far used namespaces that were pre-created when you installed OLM. In some cases, you might want to isolate your Operator deployments into a namespace that you create. For this, you will need to create a unique namespace and OperatorGroup. My namespace for this example is jeff-operators:
$ kubectl create namespace jeff-operators $ cat <<EOF | kubectl create -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: jeff-operators namespace: jeff-operators status: lastUpdated: "2020-10-07T13:44:54Z" namespaces: - "" EOF
Note that you create the unique namespace before creating your CatalogSource and subscription. You will need to create these components in the namespace that contains your OperatorGroup.
Note: For more about OperatorGroups, see Operator Multitenancy with OperatorGroups in the OLM GitHub repository.
OLM bundle automation in Operator SDK
The operator-sdk includes a new run bundle command that uses a temporary bundle index image for many of the steps described in this article. The run bundle command will be useful for developers needing to test their Operators using the OLM bundle architecture. Here's an example of the command:
$ operator-sdk run bundle quay.io/username/doo-operator-bundle:v0.0.1
Conclusion
The Operator Lifecycle Manager's bundle architecture is an advanced mechanism for describing, publishing, and deploying Operators on Kubernetes clusters. This article introduced you to using OLM's bundle deployment architecture and the Operator SDK to deploy a Kubernetes Operator.
Resources
See the following resources to learn more about OLM and the Operator Framework:
- Visit the OLM GitHub repository and OLM homepage for more about using the Operator Lifecycle Manager to install, manage, and upgrade Operators and their dependencies in a Kubernetes cluster.
- Learn more about Operator multitenancy with OperatorGroups.
- See the Guide to building a Golang based Operator using Operator SDK for a quickstart for creating Go-based Operators and more.
- Find out more about SQLite and the Database Browser for SQLite.
- Get started with gRPC and using the grpcurl command-line tool to interact with gRPC servers.
Acknowledgments
Thanks to Red Hat engineers Eric Stroczynski and Jesus Rodriguez for reviewing this article.
Last updated: February 5, 2021