Article

Using Red Hat OpenShift image streams with Kubernetes deployments

September 20, 2019
Kubernetes CI/CD Containers
Fernando Lozano
Curriculum Architect
OpenShift Operator

This article demonstrates an application update scenario which leverages Red Hat OpenShift image streams together with standard Kubernetes native resources. It also shows how image streams automatically redeploy application pods after an update to their container image.

Best of all, Kubernetes resources enhanced with OpenShift image streams are still compatible with standard Kubernetes clusters. This fact enables the use of the same resource definitions to support multiple Kubernetes distributions, and at the same time take advantage of features unique to OpenShift.

At the end of this article, we present a few considerations around using image IDs and image name tags to manage your ability to roll back to previous versions of an application.

Benefits of image streams

One of the main features that OpenShift provides over upstream Kubernetes is image stream resources. Using image streams brings many benefits, including:

  • Portability: Image streams make your pods independent of registry servers. You can copy container images from a public registry on the internet into a private registry inside your organization. There is no need to change image references or container engine configurations on your cluster nodes.
  • Consistency: Image stream names and image stream tags can follow whatever standards fit your organization. Container image names and tags follow different conventions and have different meanings depending on the vendor.
  • Reproducibility: Image streams make it easy to reference immutable container images by their image IDs. Image names and tags are mutable and could point to different images at different times. Image streams ensure that pods use a known, fixed image ID.
  • Stability: Image streams ensure that all replica pods use the same immutable image. Pods from the same replica set, that each reference images using names and tags, may end up running different container images. Users might get different results depending on which pod a service or an ingress sends each request.
  • Reversibility: If an image name and tag changes and the new container image has issues, there is no reliable way to roll back to the last known good container image. Because image streams keep a history (stream) of image IDs, generations, and time stamps, you can identify and roll back to an older, known working image if needed.
  • Automation: An image stream generates image change events that trigger the redeployment of pods from workload resources that reference the image stream. This feature allows for simple continuous deployment (CD) scenarios without requiring Jenkins and other complex tools.

OpenShift image stream basics

Image streams are named references to container images. The OpenShift extension resources reference container images indirectly, using image streams. Kubernetes standard resources reference container images directly by their registry, name, and tag.

Kubernetes users can avoid stability, reproducibility, and reversibility issues by properly managing image tags. By the end of the demonstration, you should have an understanding of how these issues may affect Kubernetes deployments, and some strategies to prevent these issue. This task can be done, regardless, but it is easier using OpenShift image streams.

Image streams are part of the OpenShift extension APIs. Other OpenShift extension resources, such as build configurations and deployment configurations, provide native support for image streams. OpenShift tooling, such as the oc command, offers easy-to-use commands to manage image stream resources, as well as other extension API resources.

OpenShift adds its extension APIs using standard Kubernetes extension mechanisms, such as custom resource definitions (CRDs) and admission plugins. This feature allows OpenShift to support using image streams together with standard Kubernetes workload API resources, such as Deployments, StatefulSets, and Jobs.

Demonstration scenario

Maybe you are not ready to switch fully to OpenShift extensions. Maybe you need to keep supporting other Kubernetes distributions. Don’t worry, you can enable image stream support for standard Kubernetes resources in a non-intrusive way. These OpenShift-enhanced resources can be used with standard Kubernetes distributions that will silently ignore the OpenShift extensions since the necessary modifications are made to these resources using annotations.

Kubernetes annotations allow adding non-identifying metadata to any Kubernetes resource. Any data stored as an annotation does not change the schema nor the semantics of a Kubernetes resource.

The fact that OpenShift uses annotations means that the same YAML file works unchanged with any standard Kubernetes cluster while taking advantage of image streams on an OpenShift cluster. When you use that YAML file with OpenShift, it processes the annotations. When you use the same YAML file with a Kubernetes distribution without image Streams, it simply works as if the annotations were not there.

Choosing test container images

To emulate an application update scenario, we need two versions of a container image. One of them is the old image, and the other is the new image. To keep things simple and be compliant with the Red Hat Enterprise Linux (RHEL) EULA, the demonstration uses the base Universal Base Image (UBI).

Start by finding the currently available tags for Red Hat's UBI base image. You can use the skopeo inspect command and, if you wish, filter the output using the jq JSON parser: 

$ skopeo inspect docker://registry.access.redhat.com/ubi8/ubi \
| jq -r '.RepoTags' -
[
  "8.0",
  "8.0-122",
  "8.0-126",
  "8.0-129",
  "8.0-154",
  "latest"
]

Smaller numbers denote older builds of the UBI base image, and higher numbers are more recent builds. For the following demonstration, the tags 8.0-122 and 8.0-154 are used.

This demonstration illustrates the method in which Red Hat manages image tags. Red Hat never overrides tags named as major-minor-build. Other tags, such as major-minor (e.g., 8.0 and latest) are floating tags. Floating tags refer to a tag that point to different container images by ID over time.

Deploying basic Kubernetes

To make sure you control container image updates during this demonstration, copy the old container image into a public registry. For example, Quay.io is used in this demonstration. You can register at Quay.io for free and publish your container images there for everyone to consume.

Use the Quay.io web interface to create a public repository on your Quay.io account. Name that repository ubi.

Log in to Quay.io using podman and copy your old UBI image there using skopeo. Replace flozanorht in the following commands with your Quay.io account, and tag the destination image as 8: 

$ podman login -u flozanorht quay.io
Password:
Login Succeeded!
$ skopeo copy docker://registry.access.redhat.com/ubi8/ubi:8.0-122 \
docker://quay.io/flozanorht/ubi:8
...
Writing manifest to image destination
Storing signatures

The following listing shows a Kubernetes Deployment resource that references the old image. This is a bare deployment on purpose and does not include important features that are typically included on runtime resources, such as health probes and resource limits, in order to focus on adding support for image streams.

Remember to change the image attribute to refer to your Quay.io account: 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  replicas: 1
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: quay.io/flozanorht/ubi:8
          command:
            - "/bin/bash"
            - "-c"
            - "while true; do ls /root/buildinfo/; sleep 30; done"

This deployment creates pods that loop forever, logging the contents of /root/buildinfo. Red Hat updates a file name under this folder with the build number of each new UBI base image. We will use this log to verify that running pods are actually using the appropriate image.

Deploying on OpenShift the Kubernetes way

Log into your OpenShift cluster, which can be either a recent release of OpenShift 3.x or a more recent release of OpenShift 4.x, and either be hosted at a cloud provider or on your local laptop using Minishift or CodeReady Containers. This demonstration should work the same way for each of these options.

I am using the OpenShift oc command-line tool, but the following steps would work the same way using the kubectl command and a standard Kubernetes cluster. I will only use features exclusive to OpenShift when I create an image stream.

Create a new project and create the Kubernetes deployment resource from the YAML file referenced previously: 

$ oc create -f myapp.yaml
deployment.apps/myapp created

After a few moments, a running pod with one container will be ready: 

$ oc get deployment
NAME    READY UP-TO-DATE   AVAILABLE AGE
myapp   1/1   1            1         6s
$ oc get pod
NAME                    READY   STATUS    RESTARTS   AGE
myapp-75c97cd8f-m5pjk   1/1     Running   0          10s

Check the pod logs, replacing the name of the pod from the above command's output, to see the build number of its container image. It should match the tag you picked up as your old image: 

$ oc logs myapp-75c97cd8f-m5pjk | tail -n1
Dockerfile-ubi8-8.0-122

Verify that the running pod references the old container image by its name and tag: 

$ oc get pod myapp-75c97cd8f-m5pjk -o jsonpath='{.spec.containers[0].image}{"\n"}'
quay.io/flozanorht/ubi:8

Creating an image stream

So far, we could be using any Kubernetes distribution and its associated kubectl command-line tool. Now we switch to the OpenShift-specific features.

Create an image stream that points to your old image on Quay.io. Also included is an image stream tag resource which points to the current image ID of the old image name and the tag it references.

Remember to change the image name on the following command to refer to your Quay.io account: 

$ oc import-image ubi --confirm --all --from quay.io/flozanorht/ubi
imagestream.image.openshift.io/ubi imported
...
$ oc get istag
NAME    IMAGE REF                                   UPDATED
ubi:8   quay.io/flozanorht/ubi@sha256:a17a...e8e1   6 seconds ago

Record the image ID as it will be referenced later: It is the string of random hexadecimal numbers after the sha256 text. This string is the real ID of an immutable container image. It will be compared with the ID you get after the container image is updated.

Adding OpenShift annotations to a Kubernetes deployment

Now starts the real fun. The oc tool provides the handy command oc set triggers that adds an image change trigger annotation to Kubernetes resources via the workloads API. This command takes the name of the image stream and the tag: 

$ oc set triggers deployment/myapp --from-image ubi:8 -c myapp
deployment.extensions/myapp triggers updated

List the deployment resource in raw YAML format to see the annotation that does the "magic:" 

$ oc get deployment myapp -o yaml | grep -A2 annotations:
  annotations:
    deployment.kubernetes.io/revision: "2"
    image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","name":"ubi:8"},"fieldPath":"spec.template.spec.containers[?(@.name==\"myapp\")].image"}]'

This annotation uses a JSONpath expression to update the image reference inside the Kubernetes resource. If you want to use image streams with other kinds of Kubernetes resources, such as cron jobs, you need to modify the JSONpath expression appropriately. If you use the oc set triggers command, OpenShift provides the expression for you.

Adding this annotation fires an image change event to make sure the currently running pods use the container ID from the image stream. After a few moments, the new pod is ready and running. Check that it now refers to the same image ID from the image stream: 

$ oc get pod
NAME                     READY STATUS RESTARTS   AGE
myapp-58cc598448-qr2xn   1/1 Running 0     10s
$ oc get pod myapp-58cc598448-qr2xn -o jsonpath='{.spec.containers[0].image}{"\n"}'
quay.io/flozanorht/ubi@sha256:a17a..e8e1

Automatic redeployment on image updates

Copy your "new" container image to Quay.io, overriding the same name and tag that referred to the old image: 

$ skopeo copy docker://registry.access.redhat.com/ubi8/ubi:8.0-154 \
docker://quay.io/flozanorht/ubi:8
...

Verify that the pod is still using the old container image ID: 

$ oc get pod myapp-58cc598448-qr2xn -o jsonpath='{.spec.containers[0].image}{"\n"}'
quay.io/flozanorht/ubi@sha256:a17a..e8e1

Update your image stream to point to the new image and verify that the image stream tag now refers to a new image ID: 

$ oc import-image ubi --confirm --all
imagestream.image.openshift.io/mysql imported
...
$ oc get istag
NAME    MAGE REF                                   UPDATED
ubi:8   quay.io/flozanorht/ubi@sha256:985e..286e   34 seconds ago

Updating the image stream triggers a redeployment. After a few moments, a new pod will be created referencing the new image ID: 

$ oc get pod
NAME                     READY  STATUS   RESTARTS   AGE
myapp-6b6c9c9787-t8kmd   1/1    Running  0          24s
$ oc get pod myapp-6b6c9c9787-t8kmd -o jsonpath='{.spec.containers[0].image}{"\n"}'
quay.io/flozanorht/ubi@sha256:985e..286e

Finally, the new pod's logs show the new container image's build number: 

$ oc logs myapp-6b6c9c9787-t8kmd | tail -n1
Dockerfile-ubi8-8.0-154

Events on the Kubernetes deployment shows both redeployments as scale up and down events of its replica set: 

$ oc describe deployment myapp
...
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  18m    deployment-controller  Scaled up replica set myapp-75c97cd8f to 1
  Normal  ScalingReplicaSet  13m    deployment-controller  Scaled up replica set myapp-58cc598448 to 1
  Normal  ScalingReplicaSet  12m    deployment-controller  Scaled down replica set myapp-75c97cd8f to 0
  Normal  ScalingReplicaSet  2m53s  deployment-controller  Scaled up replica set myapp-6b6c9c9787 to 1
  Normal  ScalingReplicaSet  2m41s  deployment-controller  Scaled down replica set myapp-58cc598448 to 0

Remember you got two redeployments: the first when adding the annotation and the second when updating the image stream. You may wonder if you had created the deployment resource already including the annotation, instead of adding it later, would this prevent one redeployment? The answer is no.

OpenShift triggers the image change event after the deployment resource is created. By that time, the Kubernetes deployment already created a replica set and a pod. The only change would be that no pods from the first deployment would ever be seen. They would be replaced by new pods before reaching the running state.

You can describe the image stream to see its history (or stream) of image IDs, in reverse chronological order: 

$ oc describe is ubi | grep -A5 tagged
  tagged from quay.io/flozanorht/ubi:8

  * quay.io/flozanorht/ubi@sha256:985e..286e
      10 minutes ago
    quay.io/flozanorht/ubi@sha256:a17a..e8e1
      10 minutes ago

You can force your image stream to use an older image ID, thus safely rolling back to a previous known-good container image.

Much more to learn about image streams

This demonstration only touched upon the basics of image streams in OpenShift. Other nice features of image streams include:

  • Image streams with multiple tags: Each tag can refer to a different registry, image names, and tags.
  • Scheduled updates to image streams: More built-in automation.
  • Image streams that reference other image streams: Multiple levels of indirection.
  • Image streams and pull secrets in shared projects: Make it easier for your team to use your enterprise registry.
  • Deploy code changes instantly: Push OpenShift builds to an image stream.

Alternatives to image streams for managing image IDs

The stability, reproducibility, and reversibility issues hinted at in the introduction are consequences of using floating container image tags. If your organization (or your vendor) manages their tags by providing a new non-floating tag for each new image, as Red Hat provides for its container images, you can rely on these tags to avoid multiple pods of your application running different container images.

Sure, it's more convenient for a developer to deploy the ubi8/ubi:8.0 image than the ubi8/ubi:8.0-122 image. Standard Kubernetes provides no automated mechanism to take a floating tag and record the image ID associated with it at different points in time. OpenShift makes using floating tags safe, thanks to image streams.

If you create your Kubernetes deployments to reference non-floating tags, you need to update the deployment resource when you update your application. You could implement these updates as part of a CI/CD pipeline using a tool like Jenkins or Tekton.

Operator-based software, such as Red Hat OpenShift Container Platform 4 cluster operators, configure their Deployments to reference image IDs and do not rely on any tag. New releases rely on the Operator Lifecycle Manager to update its deployments.

Note that using image IDs alone may not excuse you from managing non-floating tags. Your strategy depends on your registry server software. The OpenShift internal registry retains a configurable number of old image IDs for an image name. Older images will be pruned and it is not possible to roll back to them.

Red Hat Quay follows a different approach: All images whose IDs are not referenced by any tag are pruned by a background task. Registry servers that follow this approach require that you maintain non-floating tags to prevent image pruning.

Learn more

Thanks to Adan Kaplan, Andrew Block, Ben Browning, and Raffaele Spazzoli for their reviews and comments on drafts of this article.

Last updated: July 1, 2020

Comments