Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Deploying Kubernetes Operators with Operator Lifecycle Manager bundles

February 8, 2021
Jeff McCormick
Related topics:
GoKubernetesOperators

Share:

    This article shows an example of using the Operator Lifecycle Manager (OLM) bundle deployment architecture to deploy a Red Hat OpenShift or other Kubernetes Operator. You will learn how to use OLM and the Operator SDK (both components of the Kubernetes Operator Framework) together to deploy an Operator.

    About the example

    I tested the Operator Lifecycle Manager example on both an OpenShift 4.6 cluster (via Red Hat CodeReady Containers) and a local Kubernetes Kind 1.18 cluster. I used Operator SDK 1.0 and OLM 0.15.0 for this example.

    Scaffolding the Operator

    A bundle is an Operator packaging construct that contains an Operator definition and manifests used to determine how the Operator is deployed onto a Kubernetes cluster. The original OLM package manifest format has migrated to the bundle format.

    In this example, we will use Operator SDK 1.0 to generate an Operator bundle and create the Operator for deployment. Enter the following operator-sdk commands to begin scaffolding the sample Operator:

    $ operator-sdk init --domain=example.com --repo=github.com/example-inc/doo-operator
    
    $ operator-sdk create api --group cache --version v1 --kind Doo --resource=true --controller=true
    

    Working with container images in Operator SDK

    When operator-sdk generates or scaffolds an Operator, it does not include any logic about your application. You will need to include information about how to install and manage your application. Like any other Operator, the resulting code is built into a container image. In this example, the Operator image is named quay.io/username/doo-operator:v0.0.1.

    Entering the operator-sdk init command creates a project structure. This command also creates a Makefile that you can use for various build and deployment tasks. Enter the following to build an Operator image using a Makefile:

    $ make docker-build docker-push IMG=quay.io/username/doo-operator:v0.0.1
    

    Note: See the Operator SDK documentation for more about building Operators with operator-sdk and Golang.

    Building an Operator bundle image

    Apart from the Operator image, an operator bundle is an OLM-prescribed format for holding Operator metadata. The metadata contains everything Kubernetes needs to know to use the Operator, including its custom resource definitions (CRDs), required role-based access and control (RBAC) roles and bindings, dependency tree, and more.

    Generate a bundle

    You can use the operator-sdk-generated Makefile to create a bundle for your Operator:

    $ make bundle IMG=quay.io/username/doo-operator:v0.0.1
    

    This command generates an on-disk set of bundle manifests. Here is an example of the directory structure for a generated bundle:

    bundle
    
    ├── manifests
    
    │   ├── cache.example.com_dooes.yaml
    
    │   ├── doo.clusterserviceversion.yaml
    
    │   └── doo-metrics-reader_rbac.authorization.k8s.io_v1beta1_clusterrole.yaml
    
    ├── metadata
    
    │   └── annotations.yaml
    
    └── tests
    
        └── scorecard
    
            └── config.yaml
    

    The make bundle command also creates a Dockerfile (bundle.Dockerfile), which is used to build a bundle image. The bundle image is an Open Container Initiative (OCI)-compliant image that holds the generated on-disk bundle manifest and metadata files.

    Create and push the bundle image

    In this example, we'll name the Operator bundle image as follows:

    quay.io/username/doo-operator-bundle:v0.0.1
    

    To create and push the image, run the following Makefile targets:

    $ make bundle-build BUNDLE_IMG=quay.io/username/doo-operator-bundle:v0.0.1
    
    $ make docker-push IMG=quay.io/username/doo-operator-bundle:v0.0.1
    

    Building the Operator index image

    Another OLM concept is the index image. This container image serves an application programming interface (API), which describes information about your sample Operator. The index image includes information from your bundle image by running opm, an Operator registry command:

    $ opm index add --bundles quay.io/username/doo-operator-bundle:v0.0.1 --tag quay.io/username/doo-operator-index:v0.0.1
    
    $ podman push quay.io/username/doo-operator-index:v0.0.1
    

    The index image holds an SQLite database with bundle definitions. It also runs a gRPC service when the image is executed. The gRPC service lets consumers query the SQLite database about the Operators the index contains.

    Note: You can download the opm command from the Operator Framework's Operator registry.

    Running the bundle index image

    At this point, you should have OLM installed on your Kubernetes cluster. OLM is installed by default on OpenShift clusters. You can use Operator SDK's olm install command to install OLM on any other Kubernetes cluster manually.

    Once you have an index image, deploy it by creating an OLM CatalogSource resource. For our sample Operator, we create the CatalogSource as follows:

    $ cat <<EOF | kubectl create -f -
    
    kind: CatalogSource
    
    metadata:
    
      name: doo-operator
    
      namespace: operators
    
    spec:
    
      sourceType: grpc
    
      image: quay.io/username/doo-operator-index:v0.0.1
    
    EOF
    

    This CatalogSource is created in an existing namespace, created by OLM, named operators. On OpenShift clusters, this namespace is called openshift-operators. When you create the CatalogSource, it causes the index image to be executed as a pod. You can view it as follows:

    $ kubectl -n operators get pod --selector=olm.catalogSource=doo-operator
    
    NAME                                                              READY   STATUS      RESTARTS   AGE
    
    doo-operator-79x8z                                                1/1     Running     0          136m
    

    You can look at the pod's log to ensure the image is serving the gRPC API:

    $ kubectl -n operators logs pod/doo-operator-79x8z
    
    time="2020-10-05T13:17:04Z" level=info msg="Keeping server open for infinite seconds" database=/database/index.db port=50051
    
    time="2020-10-05T13:17:04Z" level=info msg="serving registry" database=/database/index.db port=50051
    

    Deploying the Operator

    We use an OLM subscription resource to trigger a specific Operator deployment. With the following command, we create a Subscription that triggers the deployment of our sample Operator:

    $ cat <<EOF | kubectl create -f -
    
    apiVersion: operators.coreos.com/v1alpha1
    
    kind: Subscription
    
    metadata:
    
      name: doo-subscription
    
      namespace: operators 
    
    spec:
    
      channel: alpha
    
      name: doo
    
      source: doo-operator
    
      sourceNamespace: operators
    
    EOF
    

    Notice that the Subscription is created in the pre-existing operators namespace so that OLM will create the sample Operator in that same namespace. (Note, again, that on OpenShift clusters, the namespace is called openshift-operators.)

    Verifying the Operator

    We can use the following commands to verify the sample Operator is running. Let's start by verifying the Subscription has been created:

    $ kubectl -n operators get subscription
    
    NAME               PACKAGE   SOURCE         CHANNEL
    
    doo-subscription   doo       doo-operator   alpha
    

    Next, verify the operator CSV has successfully deployed:

    $ kubectl -n operators get csv
    
    NAME         DISPLAY        VERSION   REPLACES   PHASE
    
    doo.v0.0.1   doo-operator   0.0.1                Succeeded
    

    Finally, verify the Operator is running:

    $ kubectl -n operators get pod
    
    NAME                                      READY   STATUS    RESTARTS   AGE
    
    doo-controller-manager-6c4bdf7db6-jcvpn   2/2     Running   0          10m
    

    Testing the Operator

    We can test the sample Operator by creating a CustomResource that the sample Operator is watching.
    Create the CustomResource in the default namespace as follows:

    $ cat <<EOF | kubectl -n default create -f -
    
    {
    
               "apiVersion": "cache.example.com/v1",
    
               "kind": "Doo",
    
               "metadata": {
    
                 "name": "doo-sample"
    
               },
    
               "spec": {
    
                 "foo": "bar"
    
               }
    
             }
    
     EOF
    

    Your sample Operator should respond to the creation of the CustomResource by checking the sample Operator log:

    $ kubectl -n operators logs pod/doo-controller-manager-6c4bdf7db6-jcvpn -c manager
    
    2020-10-05T13:29:52.175Z DEBUG controller Successfully Reconciled{"reconcilerGroup": "cache.example.com", "reconcilerKind": "Doo", "controller": "doo", "name": "doo-sample", "namespace": "default"}
    

    Create a unique namespace and OperatorGroup

    Examples so far used namespaces that were pre-created when you installed OLM. In some cases, you might want to isolate your Operator deployments into a namespace that you create. For this, you will need to create a unique namespace and OperatorGroup. My namespace for this example is jeff-operators:

    $ kubectl create namespace jeff-operators
    
    $ cat <<EOF | kubectl create -f -
    
    apiVersion: operators.coreos.com/v1
    
    kind: OperatorGroup
    
    metadata:
    
      name: jeff-operators
    
      namespace: jeff-operators
    
    status:
    
      lastUpdated: "2020-10-07T13:44:54Z"
    
      namespaces:
    
      - ""
    
    EOF
    

    Note that you create the unique namespace before creating your CatalogSource and subscription. You will need to create these components in the namespace that contains your OperatorGroup.

    Note: For more about OperatorGroups, see Operator Multitenancy with OperatorGroups in the OLM GitHub repository.

    OLM bundle automation in Operator SDK

    The operator-sdk includes a new run bundle command that uses a temporary bundle index image for many of the steps described in this article. The run bundle command will be useful for developers needing to test their Operators using the OLM bundle architecture.  Here's an example of the command:

    $ operator-sdk run bundle quay.io/username/doo-operator-bundle:v0.0.1
    

    Conclusion

    The Operator Lifecycle Manager's bundle architecture is an advanced mechanism for describing, publishing, and deploying Operators on Kubernetes clusters. This article introduced you to using OLM's bundle deployment architecture and the Operator SDK to deploy a Kubernetes Operator.

    Resources

    See the following resources to learn more about OLM and the Operator Framework:

    • Visit the OLM GitHub repository and OLM homepage for more about using the Operator Lifecycle Manager to install, manage, and upgrade Operators and their dependencies in a Kubernetes cluster.
    • Learn more about Operator multitenancy with OperatorGroups.
    • See the Guide to building a Golang based Operator using Operator SDK for a quickstart for creating Go-based Operators and more.
    • Find out more about SQLite and the Database Browser for SQLite.
    • Get started with gRPC and using the grpcurl command-line tool to interact with gRPC servers.

    Acknowledgments

    Thanks to Red Hat engineers Eric Stroczynski and Jesus Rodriguez for reviewing this article.

    Last updated: February 5, 2021

    Recent Posts

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    • How to integrate vLLM inference into your macOS and iOS apps

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue