Overhauling memory tuning in OpenJDK containers updates

We are about to ship the next update to the Universal Base Image (UBI) of the Red Hat build of OpenJDK containers. This article explains the changes in this update and the updates to come. The OpenJDK project provides new updates on a quarterly cycle and it is called the Cumulative Patch Update (or CPU for short). These typically land in January, April, July, and October. The last one shipped versions 8u362, 11.0.18, and 17.0.6 on January 17, 2023.

We release updated UBI for OpenJDK containers with the updated OpenJDK versions very soon after the upstream releases. For CPU updates, we generally limit the container changes solely to the updated OpenJDK. In order to extend and enhance the containers, we need to release updates at other times. Our aim is to make approximately quarterly container updates spaced out between the CPU releases. The next container feature update will be released by March 10, 2023.

What to expect in the next container update

The next release of the UBI9 OpenJDK containers, version 1.14, is a relatively minor update with just two user-visible changes:

1. Adjusted the way we build, which should result in smaller images (OPENJDK-1336).
2. Removed an old, long-deprecated JBoss Maven repository (OPENJDK-921).

The next release of the UBI8 OpenJDK containers, version 1.15, is more exciting. We are making a change to the default JVM memory tuning parameters, moving away from setting the JVM -Xmx parameter and instead using -XX:MaxRAMPercentage (OPENJDK-1486). 

The containers set a ceiling value on the amount of heap space that Java can allocate, defaulting to 50% and adjustable by setting JAVA_MAX_MEM_RATIO (e.g. JAVA_MAX_MEM_RATIO=80.0). Until now, our container run scripts were responsible for querying the kernel Cgroup settings, calculating what the ceiling should be as an absolute value, and setting the OpenJDK -Xmx flag accordingly.

In the times since this feature was developed, OpenJDK has grown its own container awareness and can read and respond to Cgroup memory limits. The JVM parameter -XX:MaxRAMPercentage was added. Using it in preference to -Xmx has become best practice because the effective limit can track changes to the container memory limit. Additionally, the kernel's Cgroups feature (retrospectively renamed 'Cgroups v1') was rewritten from scratch as Cgroups v2. OpenJDK 11 and 17 have support for Cgroups V2, with OpenJDK 8 support due in the forthcoming April update (8u372).

We are moving away from the container-run scripts reading the Cgroups filesystem(s) and calculating the absolute heap limit. Now we let the JVM do this instead. This also means the containers will be ready for Cgroups v2. The relative limit can still be adjusted by users from our default of 50% by setting JAVA_MAX_MEM_RATIO (to e.g. 80.0 for 80%). As always, users can override the container run parameters entirely by defining JAVA_OPTS.

Default maximum heap size

A reasonable value for JVM maximum heap size depends on the context in which the JVM is running. In the ideal container scenario, OpenJDK can expect to be the only process running and should use all of the memory made available to it. The heap usually accounts for the majority of a Java process's memory requirements, but the JVM still needs memory for things not stored on the heap.

Unfortunately, the ideal case doesn't always happen. Even on a correctly configured production container run-time environment, there may be short-lived processes started up as part of readiness probes, liveness probes, or the actions of OpenShift operators. Perhaps the Java application launches external sub-processes, and they all need a slice of RAM.

In some circumstances, containers are run without memory limits in place, such as on an OpenShift environment without default resource quotas (see OpenShift documentation for setting default resource limits) or on a developer machine during testing. In those circumstances, the effective memory limit is the total RAM accessible on the machine. However, this is being shared with all the other processes running in the operating system, so the JVM can't claim it all for itself.

For these reasons, OpenJDK set the max heap default to 25% of available memory (whether that's total physical or container limited). We believe this is very low for the container use case, leaving 3/4 of container allocated memory unusable. Our containers have long defaulted to 50% instead.

Changes coming in June

Fifty percent is still very low for the container use case, so we are moving the default limit up to 80% (OPENJDK-559). This leaves enough headroom for ephemeral processes and should not cause severe problems when run in a memory-unlimited context but will make better use of the allocated memory in an OpenShift environment with configured resource quotes in place as recommended.

We shipped the UBI9 containers in September 2022. Since they were new images and we did not have to worry about causing breaking changes to existing deployments, we went with -XX:MaxRAMPercentage and JAVA_MAX_MEM_RATIO defaulting to 80.0 (80%) from day 1. We've had no reports of problems from users or customers.

For UBI8, we are delaying the percentage change until a container update is scheduled for June to give people time to prepare and test. By this time, we have moved the default tuning parameter to -XX:MaxRAMPercentage and JDK8 will have gained full Cgroups V2 support.

What to do if things go wrong

These changes are very unlikely to cause any problems. We're moving to the best practice method for specifying the maximum heap size. We've shipped the new values in the UBI9 images since their first release. We're phasing the changes in and writing about them now to make sure people are aware of what's coming. With that said, here are some things to look at if you have problems.

To start, it's worth reading What’s new in OpenJDK's container awareness to get an understanding of the underlying technology and decisions. Here's how you can diagnose problems:

1. After the June container update changes the maximum heap default to 80%, set it back by defining JAVA_MAX_MEM_RATIO=50.0.
2. Disable all of the container's default JVM and GC tuning parameters entirely by defining JAVA_OPTS to your own choices (including an empty string).

3. Turn on container trace logging for debugging by specifying JAVA_OPTS_APPEND="-Xlog:os+container=trace" for JDK ≥ 11 or JAVA_OPTS_APPEND="-XX:+UnlockDiagnosticVMOptions -XX:+PrintContainerInfo" for JDK8.

Your feedback is always appreciated. Please leave your comments below.