This article demonstrates two common scenarios for updating Red Hat OpenShift 4: to a newer z-stream release and to a newer minor release. I include plenty of screenshots of actual updates from 4.5.4 to 4.5.17 and then to 4.6.4, so you know what to expect when you make these updates yourself.
OpenShift 4 makes the update process easy and provides a number of safety features to minimize the risk of a failed outcome. Still, updating OpenShift clusters can be scary the first time. This article shows how two perform typical update scenarios, step-by-step:
- Updating between two z-stream releases, from 4.y.z to 4.y.z+d. Note that you can skip z-stream releases during an update.
- Updating between two minor releases, from 4.y to 4.y+1. Note that you cannot skip a minor release during an update.
To use or not to use the web console
The update process, including web console pages and quirks, is mostly the same for earlier minor releases, for example a z-stream update of OpenShift 4.4. OpenShift 4.6 includes significant improvements to its web console that solve most if not all of the gotchas that I demonstrate in this article.
You can perform the entire cluster update process from the command-line interface (CLI) and automate it using Ansible playbooks, shell scripts, or whatever you like. If you need instructions about how to perform a cluster update using the CLI, please refer to the Red Hat OpenShift Container Platform product documentation and this article from the OpenShift knowledgebase.
If you have a test cluster that you were considering updating for some time and did not, for fear of the outcome, you can use the cluster now to follow the instructions from this article.
Note: Do not try to update a Red Hat CodeReady Containers (CRC) single-node cluster. CodeReady Containers disables some of the cluster operators required for a successful update in order to reduce its hardware requirements, and the OpenShift update process was not designed to work with a single supervisor node, anyway. Later releases of OpenShift Container Platform will support single-node OpenShift (SNO) clusters, which follow a different design and installation process from CodeReady Containers. SNO clusters will support updates when they become generally available.
Because the set of available updates and paths changes from time to time, you might not be able to follow the instructions in this article exactly as written. You might start from a different release, target a different final release, and have to pass through different intermediate releases. I hope to provide you with sufficient information to extrapolate to your specific scenario.
What is the current version of my OpenShift cluster?
Open your OpenShift web console, log in with cluster administrator rights, and choose Administration—>Cluster Settings. You should see a page that displays your cluster’s current update channel and OpenShift release. Mine is using the somewhat old 4.5.4 release, as shown in Figure 1. I have not updated my test cluster since it was first installed a while ago.
If I click on the pen icon beside my current update channel, which is stable-4.5, the web console allows me to choose only from other channels of the same OpenShift release (see Figure 2). That means that my current release cannot be immediately updated to the next minor release—at least not using the web console.
For now, do not change the update channel.
Why don't I see the channel for the next minor OpenShift release?
It might be possible to update directly to an OpenShift 4.6.x release from the CLI, but up to OpenShift 4.5, the web console comes with a hardcoded list of candidate update channels. The initial z-stream releases hard-code just their own update channels. Later z-stream releases add channels of the next minor release.
OpenShift 4.6 changes the offerings so that the list of available update channels becomes dynamic and shows channels from new releases as they become available. That means that an early 4.6.z release might become updatable to 4.7 without passing through intermediate 4.6.z releases, using the web console.
As I am still on 4.5 and I wish to follow the easy user interface (UI) provided by the web console, I have to first update to a newer 4.5.z release. Shame on me for leaving my cluster for so long without bug fixes and security updates! That is about to be corrected.
If I were in a hurry to update to OpenShift 4.6, or maybe to a very recent 4.5.x release, for example to get the fix for a critical bug that is affecting my production users, I might have to switch to a fast channel. Updates to OpenShift 4 are made available first in the fast channel. Only after a number of customers successfully run a given release without major incidents is an update added to the stable channel.
OpenShift update channels and upgrade paths contain many more nuances than I explain in this article. For deeper information about how Red Hat manages OpenShift update channels and releases, refer to The Ultimate Guide to OpenShift Release and Upgrade Process for Cluster Administrators.
Updating to a newer z-stream release
My test cluster is running release 4.5.4 and using the stable-4.5 channel. If I click Update now, the web console presents me a list of releases that I can update to, with the more recent releases shown first (see Figure 3).
Remember that the screenshot represents a point in time, in this case November 2020. You might see more (or fewer!) releases when you view them yourself.
The list might not include the latest release from your current channel. It includes only the ones with a direct update path from your current release. A few update cycles might take place before you get to the release you want or run out of update paths.
I select the 4.5.17 release, which is the latest as I write this article, and click Update. After a few moments, the web console starts displaying the status of my update, as shown in Figure 4.
The time to update depends on a number of factors, such as your internet bandwidth, size of the Red Hat Enterprise Linux CoreOS images and cluster operator images, number of nodes on your cluster, number of applications, their disruption budgets, and many other factors. My compact test cluster (only three nodes) took about 40 minutes.
Web console quirks while updating
During updates I saw a couple of times that the percent-complete indicator moved backward, for example from 70% to 25%. Don’t panic if that happens to you. I also saw a few temporary failures that disappeared without action from me, which I will explain later in this article. The upstream Cluster Version Operator (CVO) documentation explains why that happens; it is by design.
At some time during the update, your web console session might expire because the update process restarts web console pods to use a new container image. You might even get a “server error” from the OpenShift router. If that happens, refresh your web browser, log in again (if needed), and navigate back to Administration—>Cluster Settings to continue monitoring the progress of your cluster update.
The stated OpenShift goal of 100% availability during updates does not mean that all HTTP requests to the web console and applications are successful. During an update, nodes are rebooted, pods are recreated, containers are stopped, and new containers are started. Load balancers might not react instantly and might send an occasional request to a container or a node that is not available. The next HTTP request should work.
Is my OpenShift update done?
When the update finishes, the web console shows your new current version and might also show that more updates are available (see Figure 5).
In my example, it looks like I still am unable to get to any 4.6 channel, but I see a single 4.5 release I could update to (see Figure 6).
So, I would repeat the process I've shown in this article to update to 4.5.18 and so on, in the hope of being able eventually to update to OpenShift 4.6. (I know that I do not need further updates, and I will explain later how I know that.)
Before you update again to the next z-stream release, refresh your browser tab. Your web browser might be displaying stale data from your previous cluster version. OpenShift 4.6 solves that particular issue and requires no browser refresh.
Is my OpenShift update failing?
I did a few cluster updates without any issues, from the same test environment. But one of my attempts displayed a scary, red exclamation icon and a “Failing” status, shown in Figure 7.
Calm down and notice that the status is “fail-ing,” not “fail-ed." The update process is still running. Kubernetes clusters, applications, and operators can self-heal in a large number of scenarios. Sometimes the "Failing" message is caused by too short a timeout in the CVO. If that is the case, the message will fix itself with no action from you. And with no damage to your cluster and applications.
Do not panic, but give some time for OpenShift to heal itself. Be patient before you start collecting troubleshooting information and opening customer support tickets.
Pending cluster operators
If you click View details, as shown in Figure 7, the web console shows you detailed information about the status of each cluster operator: Which ones have already finished updating and which ones are still performing their updates. Figure 8 shows that the openshift-apiserver operator became degraded.
I hope that the message “Cluster update in progress," together with the less-alarming, blue information icon gives you some peace of mind: OpenShift is still performing the update; it has not failed at all.
Operators can become degraded for a while during updates and then recover by themselves. It is expected that, with each new OpenShift release, cluster operators become better at reporting their status and report fewer temporary failures.
Scrolling down, I can see that one of my API server pods is not running (see Figure 9). I know that this might happen when you update a Kubernetes deployment, so I decide to just give it some time to settle down.
If I suspected that something was wrong, maybe because I saw other operators with errors, I would check the operator logs and cluster events to find a hint of a real unrecoverable error.
After a few minutes, the “Failing” state switches to “Update available” and the desired cluster version, 4.5.17, becomes the current version. My update finishes successfully, as shown in Figure 10.
That “Failing” state could also switch to a new percent-complete message, as if the update process is just progressing and no failure ever happened. You might even see the installation go into a “Failing” state and then back to “Working towards” states a few times.
In the end, the changes you notice depend on how much attention you pay to the web console while your cluster is updating. Because I know the update is supposed to take time, I usually perform other tasks instead of staring at the OpenShift web console’s Cluster Settings page. If something really bad happens during my update, I expect OpenShift cluster monitoring to alert me.
Updating to a newer minor release
You might not be able to update from your current 4.y.z release to any 4.y+1 release, using the web console, prior to OpenShift 4.6. It might be necessary to update to a newer 4.y.z+d before the web console shows a channel for the next minor release. You might even have to perform multiple z-stream updates.
In my sample scenario, I had to update from OpenShift 4.5.3 to 4.5.17 to get the possibility of switching to the stable-4.6 channel. With my cluster updated to 4.5.17, and also after a web browser page refresh, the web console offered me the choice of a stable channel for the next minor release of OpenShift (see Figure 11).
To update to the next minor release, click on stable-4.6 to switch channels, click Update now, and then select any of the available 4.6.z releases as the new OpenShift release for your cluster. The update process from now on is the same as a z-stream update.
Is it too soon to update?
The first time I tried to update from OpenShift 4.5 to 4.6, by switching from the stable-4.5 to the stable-4.6 channel, I got the disappointing “Version not found” status in Figure 12. It didn't make any difference if I didn't stop at 4.5.17, but continued to update to 4.5.18 or later.
The message means that the stable-4.6 channel does not include my current version. At that time, the stable-4.6 channel did not include 4.5.18 (nor 4.5.17, for that matter).
Figure 13 shows a fully updated OpenShift 4.5 cluster, according to its current release update channel (stable-4.5), so you can compare it to the previous figure. The “Up to date” status means that the channel lists my current release only as an update path destination, and not as a source of any path. So there are no updates available currently, but an update might become available in the future.
If you find yourself in a similar scenario (that is, you switch to the stable channel of the next minor release, and find that no updates are available), Red Hat recommends that you switch back to a channel that lists your current release so you do not miss bug fixes and security updates. In my example, I would switch back to stable-4.5. Indeed, a few days later that channel got 4.5.19 and 4.5.20. But none of these releases are published to stable-4.6 as I write.
Because I tried checking only a few days after the first generally available (GA) release of OpenShift 4.6, all update paths were likely waiting for proof of stability from clusters using the fast channel.
If you find yourself in the same situation, you have two choices: Either check the stable-4.6 channel again from time to time, until you see a 4.6.z release update on the stable-4.6 channel, and revert back to the stable-4.5 after each check; or switch to the fast channel in the hope that there is an update path there.
Using the fast channel
In my case, after I switch to the fast-4.6 channel, I see that there are indeed updates available (see Figure 14).
And I could pick two releases to update from 4.5.18: 4.6.3 (not shown in Figure 15) and 4.6.4.
Figure 16 is proof that my test cluster finished its update to 4.6.4 using the fast channel. Notice the change in the look and feel of the web console. After OpenShift 4.7 is released, I will write a new article with new screenshots.
Do not be afraid of updating your cluster using a fast channel if you need an update right now. Red Hat fully supports using all releases from the fast channel in production environments. Make the decision to switch, or wait based on your needs for these updates and the potential impacts of waiting a little longer to update.
How to find which updates are available?
You might not consider it intuitive that the OpenShift web console allows you to switch to a channel when no updates are available. It makes sense once you consider that you select a channel to signal your intent of getting updates from that channel. If you pick a very new channel, as I did, there might be very few update paths. Then it is no surprise that a given release is not in any of them.
Having to switch channels and wait for intermediate cluster updates before you find whether you can update to the desired OpenShift release could be frustrating. Fortunately, there are multiple ways of checking for available updates. OpenShift 4.6 allows you to find them before performing an update. But because I am still on 4.5, I have to search outside of the web console.
One way is to use the OpenShift Update Graph. It shows a nice-looking, though sometimes confusing, graph of all update paths available for the selected channel. For example, I got the results shown in Figure 17 in mid-November 2020.
At the time I generated Figure 17, there was no update path from any 4.5.z release to 4.6 in the stable channel. When I instead selected the fast-4.6 channel, I got the much richer graph in Figure 18.
To help you visualize the state of that graph at the time I captured the screenshots for this article, I colored all 4.5.z releases with arrows getting to any 4.6.z release and zoomed in so that this part of the graph is easier to read (see Figure 19).
Figure 19 makes it easier to see that I could update from 4.5.16, 4.5.17, 4.5.18, and 4.5.19 to an OpenShift 4.6 release using the fast-4.6 channel.
Finding available updates from the shell
Another way to list update paths from an OpenShift 4 update channel comes from this article. The following example lists all updates available for 4.5.4 from the stable-4.5 channel. The last command includes a very long jq filter. Make sure the entire filter is a single shell argument. Notice the single quotes around it.
$ export CURRENT_VERSION=4.5.4
$ export CHANNEL_NAME=stable-4.5
$ curl -sH 'Accept:application/json' "https://api.openshift.com/api/upgrades_info/v1/graph?channel=${CHANNEL_NAME}" | jq -r --arg CURRENT_VERSION "${CURRENT_VERSION}" '. as $graph | $graph.nodes | map(.version=='\"$CURRENT_VERSION\"') | index(true) as $orig | $graph.edges | map(select(.[0] == $orig)[1]) | map($graph.nodes[.].version) | sort_by(.)'
[
"4.5.11",
"4.5.13",
"4.5.14",
"4.5.15",
"4.5.16",
"4.5.17",
"4.5.5",
"4.5.6",
"4.5.7",
"4.5.8",
"4.5.9",
]
All those pipe signs (|) belong to the jq filter. They are part of the fifth argument of the jq command. Do not mistake them for shell pipes! Also note that the jq filter sorts its output in a somewhat misleading way. It uses string ordering, not semantic version ordering. Notice that the most recent version, which was 4.5.17, appears in the middle of the output.
An alternative to writing that long jq filter is using the available-updates.sh script from the Cincinnati developers. The CVO uses the Cincinnati protocol to describe update channels.
Download the available-updates.sh script and make it executable. Then set the CHANNEL environment variable and pass the starting version as an argument. The following example lists what would be my options after I update to 4.5.17, using the fast-4.6 channel:
$ export CHANNEL=fast-4.6
$ ./available-updates.sh 4.5.17
4.5.18 quay.io/openshift-release-dev/ocp-release@sha256:72e3..f366 https://access.redhat.com/errata/RHBA-2020:4425
4.6.4 quay.io/openshift-release-dev/ocp-release@sha256:6681..86fc https://access.redhat.com/errata/RHBA-2020:4987
If I try the stable-4.6 channel I get an empty list, that is, [].
By viewing the output of these commands, I was able to plan for two updates starting from 4.5.4: first to 4.4.17 and then to 4.6.4 using the fast channel.
As you might guess, many developers created their own visualization tools and scripts to report OpenShift cluster updates. An example is ocp4upc.
Conclusion
OpenShift 4 updates do not have to be scary. The web console makes updates easy to do, and the underlying infrastructure provided by the CVO makes the process very reliable and dependable.
There are a few quirks in the way the web console handles updates up to OpenShift 4.5. Fortunately, OpenShift 4.6 solves most of them.
The OpenShift update process is very consistent with Kubernetes’s design patterns: You declare the desired state of your cluster and let Kubernetes converge the current state to the desired state.
For more information about how the OpenShift CVO and cluster operators handle cluster updates, please see the video Red Hat OpenShift: Cluster Upgrades and Application Operator Updates and also the excellent post from the official OpenShift blog: The Ultimate Guide to OpenShift Release and Upgrade Process for Cluster Administrators.
Acknowledgments
Thanks to Eric Rich and Mike Allmen for their reviews of drafts of this article. Also thanks to W. Trevor King and Scott Dodson for their many valuable comments improving the technical information in this article. If you find any errors and inaccuracies in this article, they are mine only despite their best efforts.
Last updated: August 15, 2022