Deploying Kubernetes Operators with Operator Lifecycle Manager bundles

This article shows an example of using the Operator Lifecycle Manager (OLM) bundle deployment architecture to deploy a Red Hat OpenShift or other Kubernetes Operator. You will learn how to use OLM and the Operator SDK (both components of the Kubernetes Operator Framework) together to deploy an Operator.

About the example

I tested the Operator Lifecycle Manager example on both an OpenShift 4.6 cluster (via Red Hat CodeReady Containers) and a local Kubernetes Kind 1.18 cluster. I used Operator SDK 1.0 and OLM 0.15.0 for this example.

Scaffolding the Operator

A bundle is an Operator packaging construct that contains an Operator definition and manifests used to determine how the Operator is deployed onto a Kubernetes cluster. The original OLM package manifest format has migrated to the bundle format.

In this example, we will use Operator SDK 1.0 to generate an Operator bundle and create the Operator for deployment. Enter the following operator-sdk commands to begin scaffolding the sample Operator:

$ operator-sdk init --domain=example.com --repo=github.com/example-inc/doo-operator

$ operator-sdk create api --group cache --version v1 --kind Doo --resource=true --controller=true

Working with container images in Operator SDK

When operator-sdk generates or scaffolds an Operator, it does not include any logic about your application. You will need to include information about how to install and manage your application. Like any other Operator, the resulting code is built into a container image. In this example, the Operator image is named quay.io/username/doo-operator:v0.0.1.

Entering the operator-sdk init command creates a project structure. This command also creates a Makefile that you can use for various build and deployment tasks. Enter the following to build an Operator image using a Makefile:

$ make docker-build docker-push IMG=quay.io/username/doo-operator:v0.0.1

Note: See the Operator SDK documentation for more about building Operators with operator-sdk and Golang.

Building an Operator bundle image

Apart from the Operator image, an operator bundle is an OLM-prescribed format for holding Operator metadata. The metadata contains everything Kubernetes needs to know to use the Operator, including its custom resource definitions (CRDs), required role-based access and control (RBAC) roles and bindings, dependency tree, and more.

Generate a bundle

You can use the operator-sdk-generated Makefile to create a bundle for your Operator:

$ make bundle IMG=quay.io/username/doo-operator:v0.0.1

This command generates an on-disk set of bundle manifests. Here is an example of the directory structure for a generated bundle:

bundle

├── manifests

│   ├── cache.example.com_dooes.yaml

│   ├── doo.clusterserviceversion.yaml

│   └── doo-metrics-reader_rbac.authorization.k8s.io_v1beta1_clusterrole.yaml

├── metadata

│   └── annotations.yaml

└── tests

    └── scorecard

        └── config.yaml

The make bundle command also creates a Dockerfile (bundle.Dockerfile), which is used to build a bundle image. The bundle image is an Open Container Initiative (OCI)-compliant image that holds the generated on-disk bundle manifest and metadata files.

Create and push the bundle image

In this example, we'll name the Operator bundle image as follows:

quay.io/username/doo-operator-bundle:v0.0.1

To create and push the image, run the following Makefile targets:

$ make bundle-build BUNDLE_IMG=quay.io/username/doo-operator-bundle:v0.0.1

$ make docker-push IMG=quay.io/username/doo-operator-bundle:v0.0.1

Building the Operator index image

Another OLM concept is the index image. This container image serves an application programming interface (API), which describes information about your sample Operator. The index image includes information from your bundle image by running opm, an Operator registry command:

$ opm index add --bundles quay.io/username/doo-operator-bundle:v0.0.1 --tag quay.io/username/doo-operator-index:v0.0.1

$ podman push quay.io/username/doo-operator-index:v0.0.1

The index image holds an SQLite database with bundle definitions. It also runs a gRPC service when the image is executed. The gRPC service lets consumers query the SQLite database about the Operators the index contains.

Note: You can download the opm command from the Operator Framework's Operator registry.

Running the bundle index image

At this point, you should have OLM installed on your Kubernetes cluster. OLM is installed by default on OpenShift clusters. You can use Operator SDK's olm install command to install OLM on any other Kubernetes cluster manually.

Once you have an index image, deploy it by creating an OLM CatalogSource resource. For our sample Operator, we create the CatalogSource as follows:

$ cat <<EOF | kubectl create -f -

kind: CatalogSource

metadata:

  name: doo-operator

  namespace: operators

spec:

  sourceType: grpc

  image: quay.io/username/doo-operator-index:v0.0.1

EOF

This CatalogSource is created in an existing namespace, created by OLM, named operators. On OpenShift clusters, this namespace is called openshift-operators. When you create the CatalogSource, it causes the index image to be executed as a pod. You can view it as follows:

$ kubectl -n operators get pod --selector=olm.catalogSource=doo-operator

NAME                                                              READY   STATUS      RESTARTS   AGE

doo-operator-79x8z                                                1/1     Running     0          136m

You can look at the pod's log to ensure the image is serving the gRPC API:

$ kubectl -n operators logs pod/doo-operator-79x8z

time="2020-10-05T13:17:04Z" level=info msg="Keeping server open for infinite seconds" database=/database/index.db port=50051

time="2020-10-05T13:17:04Z" level=info msg="serving registry" database=/database/index.db port=50051

Deploying the Operator

We use an OLM subscription resource to trigger a specific Operator deployment. With the following command, we create a Subscription that triggers the deployment of our sample Operator:

$ cat <<EOF | kubectl create -f -

apiVersion: operators.coreos.com/v1alpha1

kind: Subscription

metadata:

  name: doo-subscription

  namespace: operators 

spec:

  channel: alpha

  name: doo

  source: doo-operator

  sourceNamespace: operators

EOF

Notice that the Subscription is created in the pre-existing operators namespace so that OLM will create the sample Operator in that same namespace. (Note, again, that on OpenShift clusters, the namespace is called openshift-operators.)

Verifying the Operator

We can use the following commands to verify the sample Operator is running. Let's start by verifying the Subscription has been created:

$ kubectl -n operators get subscription

NAME               PACKAGE   SOURCE         CHANNEL

doo-subscription   doo       doo-operator   alpha

Next, verify the operator CSV has successfully deployed:

$ kubectl -n operators get csv

NAME         DISPLAY        VERSION   REPLACES   PHASE

doo.v0.0.1   doo-operator   0.0.1                Succeeded

Finally, verify the Operator is running:

$ kubectl -n operators get pod

NAME                                      READY   STATUS    RESTARTS   AGE

doo-controller-manager-6c4bdf7db6-jcvpn   2/2     Running   0          10m

Testing the Operator

We can test the sample Operator by creating a CustomResource that the sample Operator is watching.
Create the CustomResource in the default namespace as follows:

$ cat <<EOF | kubectl -n default create -f -

{

           "apiVersion": "cache.example.com/v1",

           "kind": "Doo",

           "metadata": {

             "name": "doo-sample"

           },

           "spec": {

             "foo": "bar"

           }

         }

 EOF

Your sample Operator should respond to the creation of the CustomResource by checking the sample Operator log:

$ kubectl -n operators logs pod/doo-controller-manager-6c4bdf7db6-jcvpn -c manager

2020-10-05T13:29:52.175Z DEBUG controller Successfully Reconciled{"reconcilerGroup": "cache.example.com", "reconcilerKind": "Doo", "controller": "doo", "name": "doo-sample", "namespace": "default"}

Create a unique namespace and OperatorGroup

Examples so far used namespaces that were pre-created when you installed OLM. In some cases, you might want to isolate your Operator deployments into a namespace that you create. For this, you will need to create a unique namespace and OperatorGroup. My namespace for this example is jeff-operators:

$ kubectl create namespace jeff-operators

$ cat <<EOF | kubectl create -f -

apiVersion: operators.coreos.com/v1

kind: OperatorGroup

metadata:

  name: jeff-operators

  namespace: jeff-operators

status:

  lastUpdated: "2020-10-07T13:44:54Z"

  namespaces:

  - ""

EOF

Note that you create the unique namespace before creating your CatalogSource and subscription. You will need to create these components in the namespace that contains your OperatorGroup.

Note: For more about OperatorGroups, see Operator Multitenancy with OperatorGroups in the OLM GitHub repository.

OLM bundle automation in Operator SDK

The operator-sdk includes a new run bundle command that uses a temporary bundle index image for many of the steps described in this article. The run bundle command will be useful for developers needing to test their Operators using the OLM bundle architecture.  Here's an example of the command:

$ operator-sdk run bundle quay.io/username/doo-operator-bundle:v0.0.1

Conclusion

The Operator Lifecycle Manager's bundle architecture is an advanced mechanism for describing, publishing, and deploying Operators on Kubernetes clusters. This article introduced you to using OLM's bundle deployment architecture and the Operator SDK to deploy a Kubernetes Operator.

Resources

See the following resources to learn more about OLM and the Operator Framework:

• Visit the OLM GitHub repository and OLM homepage for more about using the Operator Lifecycle Manager to install, manage, and upgrade Operators and their dependencies in a Kubernetes cluster.
• Learn more about Operator multitenancy with OperatorGroups.
• See the Guide to building a Golang based Operator using Operator SDK for a quickstart for creating Go-based Operators and more.
• Find out more about SQLite and the Database Browser for SQLite.
• Get started with gRPC and using the grpcurl command-line tool to interact with gRPC servers.

Acknowledgments

Thanks to Red Hat engineers Eric Stroczynski and Jesus Rodriguez for reviewing this article.