Metaspace setting and tuning in JDK 8+ Applications (in and outside containers)

July 19, 2024
Francisco De Melo Junior
Related topics:
JavaJava MicroservicesKubernetesRuntimes
Related products:
Red Hat build of OpenJDKRed Hat build of QuarkusRed Hat Data Grid

Since JDK 8, including upper versions, Metaspace has been set apart from the main heap section of memory and set via MetaspaceSize and MaxMetaspace flags. Objectively, this means metadata of loading classes in Java are set in the non-heap region also known as the native region of the memory (again, not inside the heap).

This Metaspace region is a change from previous versions, which had Java Metaspace as part of the heap in the so-called "Permanent Generation" and was set using the Java Virtual Machine (JVM) flag: PermSize and MaxPermSize. 

Although this discussion is very well settled, this article aims to review Metaspace, its tuning, and the consequences for container and non-container usage. It also adds troubleshooting steps in case those are needed.

Metaspace explanation

As briefly mentioned above, Metaspace was introduced on JDK 8+, therefore, in the case of OutOfMemory Exception, changed from the  Java.Lang.OutOfMemoryError: PermGen to OutOfMemoryError: Metaspace. Consequently, this is the first indication that the Java version is being used above JDK 7.

Metaspace does not occupy the heap region but rather the native space of the JVM, so it is not bounded by the Xmx setting.

Metaspace will allocate regions of memory in the so-called metaspace chunks, or metachunks inside the Virtual Spaces, which are areas of contiguous address space provided by the OS. This allocation is done on demand. More information in this solution: How does the JVM divide the Metaspace in the memory?

Also in that solution, it is explained that in JDK 11, the VM.metaspace option was introduced in Java diagnostics, which provides complete detailed information about Metaspace, including the chunks, the size, the InitialBootClassLoaderMetaspaceSize, UseCompressedClassPointers, and CompressedClassSpaceSize. Also, detailed information on the chunks:

$JDK11_HOME/bin/jcmd 27689 VM.metaspace
27689:
Total Usage ( 1490 loaders): <--- number of loaders
 Non-Class: 4312 chunks,     98.45 MB capacity,    91.95 MB ( 93%) used,     6.23 MB (  6%) free,     4.60 KB ( <1%) waste,   269.50 KB ( <1%) overhead, deallocated: 28367 blocks with 4.26 MB
     Class: 2402 chunks,     17.00 MB capacity,    13.64 MB ( 80%) used,     3.21 MB ( 19%) free,   328 bytes ( <1%) waste,   150.12 KB ( <1%) overhead, deallocated: 1753 blocks with 919.75 KB
      Both: 6714 chunks,    115.45 MB capacity,   105.60 MB ( 91%) used,     9.44 MB (  8%) free,     4.92 KB ( <1%) waste,   419.62 KB ( <1%) overhead, deallocated: 30120 blocks with 5.16 MB

Using Java Diagnostic in JDK 17 brings GC details/information about the metaspace (which on JDK 17 has a buddy algorithm) and brings more details on the Metaspace Reclaim Policy:

$ jcmd PID VM.info --->
### Metaspace
MaxMetaspaceSize: 96.00 MB <----------------------------- Max Metaspace size
CompressedClassSpaceSize: 80.00 MB <--------------------- compressed class space size == CCSS
Initial GC threshold: 32.00 MB <------------------------- initial GC threshold
Current GC threshold: 65.25 MB  <------------------------ current GC threshold
CDS: on <------------------------------------------------ Class Data Sharing 
MetaspaceReclaimPolicy: balanced <----------------------- Metaspace Reclaim Policy: balanced
- commit_granule_bytes: 65536.
- commit_granule_words: 8192.
- virtual_space_node_default_size: 1048576.
- enlarge_chunks_in_place: 1.
- new_chunks_are_fully_committed: 0.
- uncommit_free_chunks: 1.
- use_allocation_guard: 0.
- handle_deallocations: 1.

Details on how to set adequate Metaspace and MaxMetaspace

A few core points below:

  • The JVM flag: Metaspace does not set the minimal size of the Metaspace, but rather it sets the initial size of the Metaspace segment. Evidently, there is no minimal usage for metaspace given the metadata will be loaded dynamically accordingly to the classes are loaded. 
  • MaxMetaspace sets the maximum size of the Metaspace, as in the upper boundary. Crossing that boundary leads to the OutOfMemoryError: Metaspace (abbreviated as OOME Metaspace).
  • Metaspace is not the only component for native usage, 

Setting initial and max sizes inside

The question from the explanation above is: should I set the initial and max sizes? 

In terms of usage, each application will load a certain amount of classes so the use is on a case-by-case basis, however, usually even large deployments likely will use less than 1GB or 2GB of Metaspace, which would be about 1 to two billion bytes of data.

It depends on the application and the response to the OOME that crossing the max limit will cause. Let's see two scenarios for discussion below.

The scenarios will be as follows:

  • Scenario 1: The application has been developed in JDK 8+ and benchmarked.
  • Scenario 2: The application has been migrated recently and has not been benchmarked.
  • Scenario 3: The application has been developed for container and benchmarked for its usage.
  • Scenario 4: The application has not been developed for container and no consumption table is available.

Scenario 1: The application has been developed in JDK 8+ and benchmarked

In case the application was developed for JDK 8+ and metaspace was already taken into consideration in its development phase is likely the memory consumption benchmarks already take into consideration the adequate initial and max metaspace size. It might be useful not to have max sizes given the already expected allocation. Otherwise, it can be useful to set the max size to trigger an OOME: Metaspace in case of abnormality.

Scenario 2: The application has been migrated recently and has not been benchmarked

In case the application was not developed for JDK 8+ but instead was migrated, it is possible the metaspace usage was not necessarily benchmarked after its migration phase. It might be useful for the user to set initial settings and max settings to avoid unexpected usage, therefore triggering an OOME: Metaspace in case of abnormality (several OutOfMemory Exceptions can happen, Metaspace is just one of them).

As a corollary from the above, should I set the initial and max size inside a container? Let's see two scenarios for discussion below.

Scenario 3: The application has been developed for container and benchmarked for its usage

In case the application was developed for containers and metaspace was already taken into consideration in the container size, as in benchmarked already taken into consideration the adequate initial and max metaspace size, it might be useful not to have max sizes given the already expected allocation. 

Otherwise, it can be useful to set the max size to trigger an OOME: Metaspace in case of abnormality, the OOME needs to be properly handled for the information to be saved for posterior investigation because the container restart would clean any non-persisted file, being heap dump, crash file, thread dump, or VM.info.  For container deployment, in case the MaxMetaspace is set, in case the native memory balloons an OOME will be triggered and this will cause the exit of the JVM and without process, the container will exit, and subsequently the pod will restart. 

Scenario 4: The application has not been developed for container and no consumption table is available

In case the application was not necessarily developed for containers and therefore memory consumption inside didn't already take into consideration the container size, it might be adequate to set initial and max metaspace sizes. Regarding the specific value to set, it can be small to cause OOME sooner or a higher value to avoid OOME conditions. 

Either way, given an unfamiliarity with the memory usage of the application inside the container, it can be useful to set memory boundaries in the container and set initial and max sizes to trigger an OOME: Metaspace in case of abnormality. Again, the OOME needs to be properly handled for the information to be saved for posterior investigation because the container restart would clean any non-persisted file, being heap dump, crash file, thread dump, or VM.info. 

In this matter, several containers are deployed using ExitOnOutOfMemoryError JVM flag, such as JBoss EAP 7, which is not an issue, but the container will exit on the spot (usually with code 3 on the pod log) and there will be no heap saved. Depending on the scenario, in case the heap investigation is required, usage of CrashOnOutOfMemoryError over ExitOnOutOfMemoryError or sending the heap for a Persistence Volume (PV) is recommended. 

For container deployment, in case the MaxMetaspace is not set, in case the native memory balloons, which eventually would lead to the container cgroups's limit breach causing a cgroups OOM-Kill, which is not triggered/controlled by the JVM, but rather by the Red Hat OpenShift Container Platform (RHOCP) node where that container is deployed. For troubleshooting those cases, see the RHOCP node details and the SOS report, in case this was collected. 

Troubleshooting Metaspace

Tools for collaborating that can be used are:

  • GC.logs: Metaspace usage/re-caps take a full garbage collection (GC) operation so those operations will be listed on the GC logs.
  • VM.info: Includes metaspace details (even more so in JDK 17) also shared libraries. To collect VM.info see this solution and to interpret its contents see this solution. 
  • Usage of benchmarks: Benchmarks come in handy if one can associate/correlate each usage with a specific version/change in the application, as highlighted in the article How to use Java container awareness in OpenShift 4.
  • Heap dump: Heap dump can be used for heap investigations and, although native allocation is outside the heap, some hints can be found inside the heap via Querying Heap Objects (OQL), such as Native Buffer allocation used by Red Hat JBoss Enterprise Application Platform (JBoss EAP) for example. However, VM.info (with Native Tracker) can be more useful depending on the situation. 

Conclusion

JDK 8+ feature metaspace is a useful part of the native region and provides a specific region for the allocation of metadata for class loading inside the memory. Even more so, it is not controlled by the Xmx/Xms settings, so it will use the native region and its behavior differs from the previous versions.

For container usage, the heap allocation is deduced from MaxRAMPercentage, which already comes by default in OpenJDK images at 50% heap (or 80% heap), so there is no need to set Xmx, Xms, or even MaxRAM directly. The native region will be deduced as the remainder of the container memory.

Finally, benchmarking the application in scenarios such as low, medium, and high request/usage is a great tool, not just for native (off-heap) investigations but heap and container memory as well. It never hurts to log and establish guidelines for the application's CPU and memory usage. Tools such as vm.info and GC logs can also be useful not just during abnormalities but also to establish baselines.

Additional resources

To learn more, read Using Java Container awareness  and Java 17: What’s new in OpenJDK's container awareness.

For any other specific inquiries, please open a case with Red Hat support. Our global team of experts can help you with any issues.

Special thanks to Aaron Ogburn for his contributions and review of this article.