Java 17: What’s new in OpenJDK's container awareness

OpenJDK has been aware of Linux containers (such as Docker and Podman, as well as container orchestration frameworks such as Kubernetes) for some time. By container awareness, we mean that OpenJDK detects when it is running inside a container. In this article, you'll learn why container awareness is useful, what has changed recently in that area of OpenJDK, and what diagnostic options are available to help developers gain insight into how the JVM determines settings.

OpenJDK's container awareness detection uses Linux's control group (cgroup) filesystem to detect enforced resource quotas. As of this writing, OpenJDK 17, OpenJDK 11.0.16+ and OpenJDK 8u372+ are the only long-term support releases that support both cgroups v1 and cgroups v2 configurations.

OpenJDK detects whether certain resource quotas are in place when running in a container and, if so, uses those bounds for its operation. These resource limits affect, for example, the garbage collection (GC) algorithm selected by the JVM, the default size of the heap, the sizes of thread pools, and how default parallelism is determined for ForkJoinPool.

OpenJDK container awareness has been available in Java 17 and Java 11 since their respective general availability (GA) releases, and in Java 8u starting with update 8u202.

Why container awareness is important

Kubernetes and many other popular cloud orchestration systems let deployments limit container resources via CPU and memory quotas. Those limits translate into options that are passed to the container engine when containers are deployed. Container engine options, in turn, set resource limits via the Linux cgroup pseudo-filesystem. The Linux kernel ensures that when resource limits are in place via the cgroup, no process goes beyond those limits (at least not for extended periods of time).

When Java processes are deployed in such an environment, cgroup limits might be set for the deployed process. If the Java Virtual Machine does not take configured cgroup limits into account, it might risk trying to consume more resources than the operating system is willing to provide to it. The result could be the unexpected termination of the Java process.

Recent changes in OpenJDK's container awareness code

Between Java 11 and Java 17, the most prominent two additions are cgroups v2 support and container awareness in the OperatingSystemMXBean.

cgroups v2 support

Since Java 15, OpenJDK detects the cgroup version in use and detects limits according to cgroup version-specific settings. For Java 15 and onwards, OpenJDK supports cgroups v1 as well as cgroups v2 or the unified hierarchy (see JDK-8230305 for more on this).

If you run Java 11 (any version lower than 11.0.16) or Java 8 (any version lower than 8u372) on a system that has only cgroups v2, no container detection will be in place and the host values will be used instead. As explained earlier, this might yield unexpected application behavior in containerized deployments.

One quick way to show which cgroup version is in use on a system is the -XshowSettings:system option of the java launcher. (This option is Linux-specific.) Here's an example:

$ java -XshowSettings:system -version
Operating System Metrics:
    Provider: cgroupv2
    Effective CPU Count: 2
    CPU Period: 100000us
    CPU Quota: 200000us
    CPU Shares: 1024us
    List of Processors: N/A
    List of Effective Processors, 4 total:
    0 1 2 3
    List of Memory Nodes: N/A
    List of Available Memory Nodes, 1 total:
    0
    Memory Limit: 1.00G
    Memory Soft Limit: 800.00M
    Memory & Swap Limit: 1.00G

openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment 21.9 (build 17.0.2+8)
OpenJDK 64-Bit Server VM 21.9 (build 17.0.2+8, mixed mode, sharing)

Other ways to figure out the cgroup configuration in use include the VM.info jcmd utility (in the section "container (cgroup) information") or the -Xlog:os+container=debug JVM option.

If no cgroup v2 support is present—if you were working with Java 11, for example—the -XshowSettings:system output would look like this:

$ java -XshowSettings:system -version
Operating System Metrics:
    No metrics available for this platform
openjdk version "11.0.14" 2022-01-18
OpenJDK Runtime Environment 18.9 (build 11.0.14+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.14+9, mixed mode, sharing)

If no system metrics are detected, the JVM process falls back to using host operating system settings.

OperatingSystemMXBean container awareness

Since Java 14, the OperatingSystemMXBean uses the JDK's internal, Linux-specific Metrics Java API to report system information. That means if cgroup limits are in place, the OperatingSystemMXBean reports those limits (as appropriate) over the container host system resources. This feature has also been backported to Java 8 (8u272 and newer) and Java 11 (11.0.9 and newer).

Note: Container awareness in OpenJDK can be disabled with the in the -XX:-UseContainerSupport JVM option. This, in turn, would disable container awareness of OperatingSystemMXBean.

The following file, named CheckOperatingSystemMXBean.java, displays information about the system on which it is running. As it is using the container-aware OperatingSystemMXBean, it will show information about either the physical host or the container resources, depending on the environment in which it is running:

import com.sun.management.OperatingSystemMXBean;
import java.lang.management.ManagementFactory;

public class CheckOperatingSystemMXBean {

    public static void main(String[] args) {
        System.out.println("Checking OperatingSystemMXBean");

        OperatingSystemMXBean osBean = (OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
        System.out.println(String.format("Runtime.availableProcessors: %d", Runtime.getRuntime().availableProcessors()));
        System.out.println(String.format("OperatingSystemMXBean.getAvailableProcessors: %d", osBean.getAvailableProcessors()));
        System.out.println(String.format("OperatingSystemMXBean.getTotalPhysicalMemorySize: %d", osBean.getTotalPhysicalMemorySize()));
        System.out.println(String.format("OperatingSystemMXBean.getFreePhysicalMemorySize: %d", osBean.getFreePhysicalMemorySize()));
        System.out.println(String.format("OperatingSystemMXBean.getTotalSwapSpaceSize: %d", osBean.getTotalSwapSpaceSize()));
        System.out.println(String.format("OperatingSystemMXBean.getFreeSwapSpaceSize: %d", osBean.getFreeSwapSpaceSize()));
        System.out.println(String.format("OperatingSystemMXBean.getSystemCpuLoad: %f", osBean.getSystemCpuLoad()));
    }

}

Compiling and running the program displays the resources available to it:

$ ./jdk-17.0.1+12/bin/javac CheckOperatingSystemMXBean.java
$ sudo podman run -ti --rm --memory 300m --memory-swap 300m --cpu-period 100000 --cpu-quota 200000 -v $(pwd):/opt:z fedora:35
[root@7a0de39d8430 opt]# /opt/jdk-17.0.1+12/bin/java CheckOperatingSystemMXBean
Checking OperatingSystemMXBean
Runtime.availableProcessors: 2
OperatingSystemMXBean.getAvailableProcessors: 2
OperatingSystemMXBean.getTotalPhysicalMemorySize: 314572800
OperatingSystemMXBean.getFreePhysicalMemorySize: 291680256
OperatingSystemMXBean.getTotalSwapSpaceSize: 0
OperatingSystemMXBean.getFreeSwapSpaceSize: 0
OperatingSystemMXBean.getSystemCpuLoad: 0.050386
[root@7a0de39d8430 opt]# /opt/jdk-17.0.1+12/bin/java -XX:-UseContainerSupport CheckOperatingSystemMXBean
Checking OperatingSystemMXBean
Runtime.availableProcessors: 4
OperatingSystemMXBean.getAvailableProcessors: 4
OperatingSystemMXBean.getTotalPhysicalMemorySize: 5028548608
OperatingSystemMXBean.getFreePhysicalMemorySize: 3474866176
OperatingSystemMXBean.getTotalSwapSpaceSize: 5027917824
OperatingSystemMXBean.getFreeSwapSpaceSize: 5027917824
OperatingSystemMXBean.getSystemCpuLoad: 0.000000

Tuning defaults for containers

In some cases, the OpenJDK default settings for memory and CPU usage might not be the desired settings for applications running in containers. OpenJDK needs to consider multi-user desktop and server systems as well as container use cases, among other things. A container is different from a desktop or server because quite often the Java process is the only one running in that container. For example, in a Kubernetes container with a memory limit of 800MB RAM, a default -XX:MaxRAMPercentage=25, probably doesn't make as much sense as it would on a multi-user desktop system, because the maximum heap size would be bound above by 200MB RAM (one-quarter of 800MB) of that 800MB container.

To tune OpenJDK for the typical container use case, options have been introduced with JDK-8186248 (and in OpenJDK 8u with JDK-8146115) that specifically allow you to set heap sizes in percentages of available (container) memory to better fit this specific use case. These options are -XX:InitialRAMPercentage, -XX:MaxRAMPercentage, and -XX:MinRAMPercentage.

Setting these percentage options when the application runs in a container is preferable to setting maximum and minimum heap size for your application via -Xmx and -Xms, respectively. The -XX options set the heap size relative to the container memory limits and get updated automatically on redeployment should those limits change in the deployment configuration. When both types of settings are in place, -Xmx and -Xms take precedence.

Another important option for overriding CPU settings when running inside containers is -XX:ActiveProcessorCount. This option lets you specify exactly how many CPU cores the JVM should use regardless of container detection heuristics.

Container detection support can also be disabled entirely using the -XX:-UseContainerSupport option.

Table 1 summarizes some useful options for tuning JVM settings when running in a container.

Opinionated configuration

Default container detection heuristics for CPU resource limits are largely modeled on the way popular container orchestration frameworks—specifically Kubernetes and Mesos—spawn containers. For example, in a Kubernetes setup, there are four (main) cases to consider when CPU resource limits are in place. There are actually even more possibilities because cluster defaults might be in place, but those are largely also covered by these cases:

1. Both spec.containers[].resources.limits.cpu and spec.containers[].resources.requests.cpu are explicitly set.
2. Only spec.containers[].resources.limits.cpu is explicitly set. Kubernetes sets spec.containers[].resources.requests.cpu to the same value as spec.containers[].resources.limits.cpu.
3. Only spec.containers[].resources.requests.cpu is explicitly set. Kubernetes sets spec.containers[].resources.limits.cpu to a value not smaller than spec.containers[].resources.requests.cpu.
4. Neither spec.containers[].resources.limits.cpu nor spec.containers[].resources.requests.cpu is set. Kubernetes keeps spec.containers[].resources.limits.cpu unset and sets the containers' CpuShares value to 2 if no other defaults are in place.

Container orchestration frameworks usually multiply the millicore value of spec.containers[].resources.requests.cpu by 1024, which then directly translates to the value set in Docker or Podman by the --cpu-shares command-line option. Therefore, the JVM will calculate the CPU shares value based on this knowledge.

In addition, the JVM will set a lower bound on a CPU core value of 1. That is, a container with a setting of spec.containers[].resources.requests.cpu=500m makes the JVM use a single CPU core (0.5 * 1024 = 512, generating an option of --cpu-shares=512; cpu-shares < 1024 results in one core). A setting of spec.containers[].resources.requests.cpu=2 makes the JVM use two CPU cores, and so on.

Note that these rules cause the JVM to think it can use only one CPU core for the final case in the list, where neither option is explicitly set. In such a case, it is recommended that you override the desired CPU core value via -XX:ActiveProcessorCount.

Note: In versions 18.0.2+, 17.0.5+ and 11.0.17+, OpenJDK will no longer take CPU shares settings into account for its calculation of available CPU cores. See JDK-8281181 for details.

The spec.containers[].resources.limits.cpu (L) millicore value directly translates to Docker's and Podman's --cpu-quota (Q) and --cpu-period (P) values. The JVM will calculate the limit—as set by (L)—based on the formula ceil(Q/P). Note that if both spec.containers[].resources.limits.cpu and spec.containers[].resources.requests.cpu are specified, the limits value takes precedence. This will make the JVM use reasonable values for CPU cores in the first three cases. To prefer shares over CPU quota, specify the -XX:-PreferContainerQuotaForCPUCount option (see JDK-8197867 for more on this).

Similarly, resource limits for RAM exist via container orchestration frameworks. spec.containers[].resources.limits.memory translates to the –memory and –memory-swap command-line options of container engines. spec.containers[].resources.requests.memory usually doesn't have an effect on the spawned containers. On nodes using cgroup v2, memory.min or memory.low might be set accordingly. Memory request settings have no effect on the JVM side other than reporting those values for diagnostics.

Diagnostic options for debugging

Trace logs can be quite useful for helping you better understand what OpenJDK's container detection logic is doing. Note that there are two different implementations of the detection logic: One for the JVM (libjvm.so) and another implemented in Java for use of core libraries.

The JVM's container detection logic is integrated with the unified logging framework and can be traced for example via -Xlog:os+container=trace. For OpenJDK 8u JVMs, the rough equivalent is -XX:+UnlockDiagnosticVMOptions -XX:+PrintContainerInfo. These traces print whether or not container detection is actually working and what values the JVM is determining to be in place by inspecting the cgroup pseudo filesystem of a deployed application.

For Java 11+ it's also useful to know which GC is being used, and you can display this information via -Xlog:gc=info. For example, when container limits allow only a single CPU to be active, the Serial GC will be selected. If more than one CPU is active and sufficient memory (at least 2GB) is allocated to the container, the G1 GC will be selected in Java 11 and later versions:

$ java -XX:ActiveProcessorCount=1 -Xlog:gc=info -version
[0.048s][info][gc] Using Serial
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment 21.9 (build 17.0.1+12)
OpenJDK 64-Bit Server VM 21.9 (build 17.0.1+12, mixed mode, sharing)

$ java -Xlog:gc=info -version
[0.006s][info][gc] Using G1
openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment 21.9 (build 17.0.1+12)
OpenJDK 64-Bit Server VM 21.9 (build 17.0.1+12, mixed mode, sharing)

Other options include the VM.info jcmd utility, which is useful for determining the container detection settings of an already running Java process. The cgroup information shown via VM.info is the same information recorded in hs_err*.log files when the JVM crashes.

Note: Most of this information is also available via JDK Flight Recorder (JFR), a powerful tool for diagnosing issues. In containers, JFR information can be accessed via Cryostat.

Conclusion

For more information on how the Kubernetes deployment configuration can have an effect on OpenJDK's selection of a GC algorithm on a cgroups v2 system, check out my screencast on this topic:

The screencast demonstrates that OpenJDK might behave slightly differently depending on your application's deployment in a container. With the help of this article, you can now make more sense of why. Feel free to tune your application's container settings so as to get the most out of your cloud deployment. Keep in mind that cgroup v2 support is only available in Java 17+ and OpenJDK 11.0.16+.

If you're interested in learning more about the ins and outs of fine-tuning Java in container-based environments, read the first article in this series, Java in single-core containers, or check out Red Hat Developer's recent series on Cryostat. You can also watch Java and containers: What's there to think about?, a DevNation Tech Talk from Christine Flood and Edson Yanaga.