Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Eliminate downtime during OpenShift rolling updates

June 8, 2022
Rupesh Patel
Related topics:
Kubernetes
Related products:
Red Hat OpenShift Container Platform

Share:

    Do your clients complain about interruptions during software upgrades? Do you observe connection failures or timeouts during those upgrades? In this article, you'll learn how you can minimize the impacts on your client visiting your services hosted on the Red Hat OpenShift Container Platform during software updates.

    A rolling update creates new pods running the new software and terminates old ones. The deployment controller performs this rollout incrementally, ensuring that a certain number of new pods are ready before the controller deletes the old pods that the new pods are replacing. For details, see Rolling strategy in the Red Hat OpenShift Container Platform documentation.

    Figure 1 shows a typical sequence of events during an update. The important point, for the purposes of this article, is that pods go through a transitional period where they are present but not functional. Achieving a zero-downtime rollout requires some care to drain traffic from the old pods and allow the OpenShift router time to update its configuration before the deployment controller removes the old pods.

    A rolling update adds and removes pods, while the service points to both old and new pods.
    Figure 1. A rolling update adds and removes pods, while the service points to both old and new pods.
    Figure 1: A rolling update adds and removes pods.

    Pod termination starts with setting its deletionTimestamp field to a non-null value to indicate that it has been marked for deletion. An oc get or kubectl get command shows such a pod in a Terminating state. A pod may exist in this state for some period of time (several seconds or minutes, possibly even hours or days) before the pod is actually removed. See Termination of Pods in the Kubernetes documentation for details.

    When a pod enters the Terminating state, different parts of the system react to resolve the transitional status:

    1. The kubelet updates the pod's status to Ready=False.

    2. The endpoint slice controller observes the update to the pod's status and removes the pod's IP address from any EndpointSlice object that has it.

    3. The OpenShift router observes this update to the EndpointSlice. The router removes the pod's IP address from the HAProxy configuration to stop HAProxy from forwarding requests to the pod. Finally, the router reloads HAProxy so that the configuration changes take effect.

    Thus, there can be a delay between when the pod is marked for deletion and when the OpenShift router reloads HAProxy with the updated configuration.

    How does this affect the risk of downtime? Suppose you have a route with some backend pod, and a client sends a request for that route. Any of the following can happen with that request:

    • HAProxy forwards the request to the backend pod, and it remains responsive for the duration of the transaction. In this case, the pod sends a response, and HAProxy forwards the response to the client. Everything is fine.

    • HAProxy forwards the request to the backend pod, and the pod is terminated during the transaction. In this case, HAProxy returns an error response to the client. This makes the service appear to be down, even though many other pods are running.

    • HAProxy forwards the request to a backend pod that has already been terminated. In this case, the connection to the pod fails. Then:

      • If there is no other backend pod, HAProxy returns an error response to the client.

      • If there is another backend pod, HAProxy retries the request with that pod. In this case, the client gets a successful response, although it might be delayed while HAProxy's connection to the first backend pod fails and HAProxy retries the request with the other pod.

    Solution: A PreStop container hook

    The risk of downtime can be almost completely eliminated through a simple solution: the introduction of an arbitrary delay during the Terminating state so that a pod continues to accept and handle requests until HAProxy stops forwarding requests to that pod. This grace period can be added by adding a PreStop hook to the deployment.

    The PreStop hook simply delays pod termination in order to allow HAProxy to stop forwarding requests to it. In addition, if the application handles long-lived connections, the PreStop hook must delay the pod's removal long enough for these connections to finish.

    Note: The application process itself may have a built-in termination grace period. In this case, adding a PreStop hook would be superfluous.

    If the application doesn't have long-lived connections, 15 to 30 seconds should be plenty of time for the PreStop hook. The administrator should test the PreStop hook with different values and set a value that suits their environment. It is crucial that the pod continues to respond to requests while it is in the Terminating state.

    Note: The administrator should keep in mind that adding a PreStop hook consumes more time for recycling pods than usual.

    Conclusion

    Graceful termination requires time. Rolling updates can take up to several minutes to complete. For certain applications, graceful termination doesn't provide value. Determining whether it is worthwhile, and how long the grace period needs to be to allow traffic to drain, is up to the administrator. When configured appropriately, graceful termination can improve the experience for your end users.

    Last updated: September 20, 2023

    Recent Posts

    • Assessing AI for OpenShift operations: Advanced configurations

    • OpenShift Lightspeed: Assessing AI for OpenShift operations

    • OpenShift Data Foundation and HashiCorp Vault securing data

    • Axolotl meets LLM Compressor: Fast, sparse, open

    • What’s new for developers in Red Hat OpenShift 4.19

    What’s up next?

    The microservice architectural approach is more than just about technology: It reaches into the foundation of your organization to allow you to build truly scalable, adaptive, complex systems that help a business adapt to rapidly changing competitive markets. In Microservices for Java Developers, you'll get a hands-on introduction to frameworks and containers through a handful of familiar patterns.

    Get the free e-book
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue