To configure a multi-container application that has dependencies within a single pod, you should control the order in which they start up or shut down to prevent race conditions (e.g., if container A is required to start after container B, or container A needs to delay termination after container B). A classic use case is the Istio proxy, a sidecar container that all traffic traverses through from the main containers. It should start prior to the main containers accepting traffic and terminated after the primary containers drain all remaining traffic.
In this article, I will take a deeper look into container startup and termination order as a part of the pod lifecycle explained in the Kubernetes upstream documentation. This topic is not brand new. However, it is worth checking out the latest Red Hat OpenShift release notes for any changes.
The container's lifetime in a pod
You can organize a normal pod with three container types, each having a lifetime (Figure 1).
Be aware that the starting order is based on the creation time of each container. Conversely, the termination order is based on issuing the order of the SIGTERM signal to each container. The following table illustrates the order in which each container type starts and stops.
Type | Starting order | Termination order | Description |
Init containers | Sequential, one by one in the order defined | Sequential, one by one in the order defined | Executed and completed one after the other before the main containers run |
Sidecar containers | Sequential, one by one in the order defined | In the opposite order, after all main containers stop | Run along with the main containers after executed from the Initialization phase |
Main containers | Sequential without blocking the next | Random, due to the time it takes for the termination process to complete | Run main application processes here |
Next, let’s look into the details of how each container type starts and stops.
Init containers
Init containers must start up successfully before the next one starts. This specification makes the starting and termination order sequential. Typical use cases for init containers are for any pre-processes that need to take place prior to running the main containers.
For instance, all the init containers in the following pod A case would start and terminate A, B, and C in order, and each init container must terminate successfully as a one-off task before the main containers start. If one of the init containers fails, the next one in the list will not start. Then the pod enters an error state, such as Init: Error and Init: CrashLoopBackOff. Examine the pod B case in Figure 2 where this situation is depicted. Init container C did not start after init container B failed. In addition, the main container did not start.
Sidecar containers
Recent versions of Kubernetes as of 1.29 (Red Hat OpenShift 4.16) added the sidecar container type. Both init containers and sidecar containers have similar configurations. However, a key difference between the two is that sidecar containers will set restartPolicy: Always so the container will keep running after the main containers start. This behavior is useful for supporting sidecar-based architectures. The starting order is sequential. If one has not started successfully, the next container will not start, as shown in the prior init containers section. Even if you mix other init containers together, they start in the order specified in the spec.initContainers array in the pod manifest.
Take a look at Figure 3. Sidecar containers A, B, and C are each started one by one and then run along with the main containers. If any sidecar container or init container fails, the main containers will not start.
Next the termination phase depicted in Figure 4 shows the value of the sidecar containers.
All sidecar containers terminate in the opposite order they started after terminating all the main containers.
This is helpful in resolving issues of any dependencies between sidecar containers and main containers (e.g., if a logging agent must be terminated after the main application so application logs are processed without any loss). If you’re interested in how this termination order is implemented, refer to this code sample.
Main containers
The primary content of your applications are run as the main containers within a pod, and it starts sequentially in order of the spec.containers array in the pod template. The kubelet starts all main containers without checking the status of each container. This is a different behavior than init and sidecar containers as previously mentioned. In other words, the next main container launches regardless of the previously started containers starting status.
This logic is quickly processed, so it can appear that all main containers started at the same time. If you want to start all main containers one by one like init containers, you can implement this approach using the postStart lifecycle hook that blocks the start of the next container until the hook is completed.
Figure 5 illustrates this process.
One interesting point to note is that all the main containers are terminated randomly in parallel (see Figure 6). As illustrated in this code sample, the termination is invoked concurrently through the Go routine for each of the containers. If you want to implement a specific termination order in the main containers, you can consider configuring preStop lifecycle hooks.
You can defer the time to issue the SIGTERM signal to the container until the hook is completed. Keep in mind that all the preStop hooks are triggered immediately in parallel as soon as the pod is in terminating status, not one by one for each of the containers. You can find the related code on GitHub.
An exception to the termination order
An additional consideration is that there is an exception to the termination order when the pod expires as a result of the terminationGracePeriodSeconds timeout. This timeout starts to count down as soon as the pod enters the terminating status. If any of the main containers were not terminated within this timeframe, the kubelet forcibly kills the container with the SIGKILL signal.
Consider the situation where the sidecar containers are running together at that time. What will occur? The sidecar containers will terminate randomly in parallel after terminating all main containers without keeping the expected reverse order. If any of the sidecar containers do not terminate immediately, the kubelet also sends the container kill signal again after waiting an additional 2 seconds.
You need to modify the terminationGracePeriodSeconds to keep the expected termination to suit this use case. For more details, refer to the documentation for the Termination of Pods and Pod shutdown and sidecar containers. The diagram in Figure 7 illustrates this process.
Final thoughts
While the functionality surrounding the lifecycle of a pod was a core capability of Kubernetes for many releases, understanding how it functions enables greater insight and the ability to fine-tune the configurations of your applications and use cases. We provided examples that included real-world use cases you can use within your own environment to harness the true power of Red Hat OpenShift and Kubernetes.
