Metaspace setting and tuning in JDK 8+ Applications (in and outside containers)

Since JDK 8, including upper versions, Metaspace has been set apart from the main heap section of memory and defined via MetaspaceSize and MaxMetaspace flags. Objectively, this means metadata of loading classes in Java is located in the non-heap region also known as the native region of the memory (again, not inside the heap).

This Metaspace region is a change from previous versions, which had Java Metaspace as part of the heap in the so-called "Permanent Generation" and was set using the Java Virtual Machine (JVM) flag: PermSize and MaxPermSize. 

Although this discussion is very well settled, this article aims to review Metaspace, its tuning, and the consequences for container and non-container usage. It also adds troubleshooting steps in case those are needed.

This article complements the previous this other one: How to use Java container awareness in OpenShift 4 providing details for the off-heap section Metaspace. This is a brief introduction to this subject, not a deep dive.

Metaspace explanation

As briefly mentioned above, Metaspace was introduced on JDK 8+, therefore, in the case of OutOfMemory Exception, changed from the  Java.Lang.OutOfMemoryError: PermGen to OutOfMemoryError: Metaspace. Consequently, this is the first indication that the Java version is being used above JDK 7.

Metaspace does not occupy the heap region but rather the native space of the JVM, so it is not bounded by the Xmx setting.

Metaspace will allocate regions of memory in the so-called metaspace chunks, or metachunks inside the Virtual Spaces, which are areas of contiguous address space provided by the OS. This allocation is done on demand. More information in this solution: How does the JVM divide the Metaspace in the memory?

Also in that solution, it is explained that in JDK 11, the VM.metaspace option was introduced in Java diagnostics, which provides complete detailed information about Metaspace, including the chunks, the size, the InitialBootClassLoaderMetaspaceSize, UseCompressedClassPointers, and CompressedClassSpaceSize. Also, detailed information on the chunks:

$JDK11_HOME/bin/jcmd 27689 VM.metaspace
27689:
Total Usage ( 1490 loaders): <--- number of loaders
 Non-Class: 4312 chunks,     98.45 MB capacity,    91.95 MB ( 93%) used,     6.23 MB (  6%) free,     4.60 KB ( <1%) waste,   269.50 KB ( <1%) overhead, deallocated: 28367 blocks with 4.26 MB
     Class: 2402 chunks,     17.00 MB capacity,    13.64 MB ( 80%) used,     3.21 MB ( 19%) free,   328 bytes ( <1%) waste,   150.12 KB ( <1%) overhead, deallocated: 1753 blocks with 919.75 KB
      Both: 6714 chunks,    115.45 MB capacity,   105.60 MB ( 91%) used,     9.44 MB (  8%) free,     4.92 KB ( <1%) waste,   419.62 KB ( <1%) overhead, deallocated: 30120 blocks with 5.16 MB

Using Java Diagnostic in JDK 17 brings GC details/information about the metaspace (which on JDK 17 has a buddy algorithm) and brings more details on the Metaspace Reclaim Policy:

$ jcmd PID VM.info --->
### Metaspace
MaxMetaspaceSize: 96.00 MB <----------------------------- Max Metaspace size
CompressedClassSpaceSize: 80.00 MB <--------------------- compressed class space size == CCSS
Initial GC threshold: 32.00 MB <------------------------- initial GC threshold
Current GC threshold: 65.25 MB  <------------------------ current GC threshold
CDS: on <------------------------------------------------ Class Data Sharing 
MetaspaceReclaimPolicy: balanced <----------------------- Metaspace Reclaim Policy: balanced
- commit_granule_bytes: 65536.
- commit_granule_words: 8192.
- virtual_space_node_default_size: 1048576.
- enlarge_chunks_in_place: 1.
- new_chunks_are_fully_committed: 0.
- uncommit_free_chunks: 1.
- use_allocation_guard: 0.
- handle_deallocations: 1.

Below OpenJDK 17 VM.metaspace:

sh-5.1$ jcmd 150 VM.metaspace
150:
Total Usage - 432 loaders, 12819 classes (1420 shared):
  Non-Class: 2103 chunks,     51.69 MB capacity,   51.31 MB (>99%) committed,    50.98 MB ( 99%) used,   338.89 KB ( <1%) free,     2.15 KB ( <1%) waste , deallocated: 87 blocks with 14.46 KB
      Class:  818 chunks,      7.62 MB capacity,    7.55 MB (>99%) committed,     7.32 MB ( 96%) used,   234.30 KB (  3%) free,   152 bytes ( <1%) waste , deallocated: 348 blocks with 84.27 KB
       Both: 2921 chunks,     59.30 MB capacity,   58.87 MB (>99%) committed,    58.30 MB ( 98%) used,   573.19 KB ( <1%) free,     2.30 KB ( <1%) waste , deallocated: 435 blocks with 98.73 KB

Virtual space:
  Non-class space:       64.00 MB reserved,      51.31 MB ( 80%) committed,  1 nodes.
      Class space:       80.00 MB reserved,       7.56 MB (  9%) committed,  1 nodes.
             Both:      144.00 MB reserved,      58.88 MB ( 41%) committed. 
Chunk freelists:
   Non-Class:
 16m: (none)
  8m:    2, capacity=16.00 MB, committed=0 bytes (  0%)
  4m:    2, capacity=8.00 MB, committed=0 bytes (  0%)
  2m: (none)
...
Total word size: 33.01 MB, committed: 0 bytes (  0%)
Waste (unused committed space):(percentages refer to total committed size 58.88 MB):
        Waste in chunks in use:      2.30 KB ( <1%)
        Free in chunks in use:    573.19 KB ( <1%)
                In free chunks:      0 bytes (  0%)
Deallocated from chunks in use:     98.73 KB ( <1%) (435 blocks)
                       -total-:    674.22 KB (  1%)

chunk header pool: 2930 items, 207.36 KB.
Internal statistics:

num_allocs_failed_limit: 6.
num_arena_births: 866.
num_arena_deaths: 2.
num_vsnodes_births: 2.
num_vsnodes_deaths: 0.
num_space_committed: 942.
num_space_uncommitted: 0.
num_chunks_returned_to_freelist: 8.
num_chunks_taken_from_freelist: 2924.
num_chunk_merges: 6.
num_chunk_splits: 2075.
num_chunks_enlarged: 1621.
num_inconsistent_stats: 0.

Settings: <--- will be also in VM.info
MaxMetaspaceSize: 96.00 MB
CompressedClassSpaceSize: 80.00 MB
Initial GC threshold: 32.00 MB
Current GC threshold: 89.00 MB
CDS: on
 - commit_granule_bytes: 65536.
 - commit_granule_words: 8192.
 - virtual_space_node_default_size: 8388608.
 - enlarge_chunks_in_place: 1.
 - use_allocation_guard: 0.

Details on how to set adequate Metaspace and MaxMetaspace

A few core points below:

• The JVM flag: Metaspace does not set the minimal size of the Metaspace, but rather it sets the initial size of the Metaspace segment. Evidently, there is no minimal usage for metaspace given the metadata will be loaded dynamically accordingly to the classes are loaded. 
• MaxMetaspace sets the maximum size of the Metaspace, as in the upper boundary. Crossing that boundary leads to the OutOfMemoryError: Metaspace (abbreviated as OOME Metaspace).
• Metaspace is not the only component for native usage, 

Setting initial and max sizes inside containers

The question from the explanation above is: should I set the initial and max sizes? 

In terms of usage, each application will load a certain amount of classes so the use is on a case-by-case basis, however, usually even large deployments likely will use less than 1GB or 2GB of Metaspace, which would be about 1 to two billion bytes of data.

It depends on the application and the response to the OOME that crossing the max limit will cause. Let's see two scenarios for discussion below.

The scenarios will be as follows:

• Scenario 1: The application has been developed in JDK 8+ and benchmarked.
• Scenario 2: The application has been migrated recently and has not been benchmarked.
• Scenario 3: The application has been developed for container and benchmarked for its usage.
• Scenario 4: The application has not been developed for container and no consumption table is available.

Scenario 1: The application has been developed in JDK 8+ and benchmarked

In case the application was developed for JDK 8+ and metaspace was already taken into consideration in its development phase is likely the memory consumption benchmarks already take into consideration the adequate initial and max metaspace size. It might be useful not to have max sizes given the already expected allocation. Otherwise, it can be useful to set the max size to trigger an OOME: Metaspace in case of abnormality.

Scenario 2: The application has been migrated recently and has not been benchmarked

In case the application was not developed for JDK 8+ but instead was migrated, it is possible the metaspace usage was not necessarily benchmarked after its migration phase. It might be useful for the user to set initial settings and max settings to avoid unexpected usage, therefore triggering an OOME: Metaspace in case of abnormality (several OutOfMemory Exceptions can happen, Metaspace is just one of them).

As a corollary from the above, should I set the initial and max size inside a container? Let's see two scenarios for discussion below.

Scenario 3: The application has been developed for container and benchmarked for its usage

In case the application was developed for containers and metaspace was already taken into consideration in the container size, as in benchmarked already taken into consideration the adequate initial and max metaspace size, it might be useful not to have max sizes given the already expected allocation. 

Otherwise, it can be useful to set the max size to trigger an OOME: Metaspace in case of abnormality, the OOME needs to be properly handled for the information to be saved for posterior investigation because the container restart would clean any non-persisted file, being heap dump, crash file, thread dump, or VM.info.  For container deployment, in case the MaxMetaspace is set, in case the native memory balloons an OOME will be triggered and this will cause the exit of the JVM and without process, the container will exit, and subsequently the pod will restart. 

Scenario 4: The application has not been developed for container and no consumption table is available

In case the application was not necessarily developed for containers and therefore memory consumption inside didn't already take into consideration the container size, it might be adequate to set initial and max metaspace sizes. Regarding the specific value to set, it can be small to cause OOME sooner or a higher value to avoid OOME conditions. 

Either way, given an unfamiliarity with the memory usage of the application inside the container, it can be useful to set memory boundaries in the container and set initial and max sizes to trigger OOME: Metaspace in case of abnormality. Again, the OOME needs to be properly handled for the information to be saved for posterior investigation because the container restart would clean any non-persisted file, being heap dump, crash file, thread dump, or VM.info. 

In this matter, several containers are deployed using ExitOnOutOfMemoryError JVM flag, such as JBoss EAP 7, which is not an issue, but the container will exit on the spot (usually with code 3 on the pod log) and there will be no heap saved. Depending on the scenario, in case the heap investigation is required, usage of CrashOnOutOfMemoryError over ExitOnOutOfMemoryError or sending the heap for a Persistence Volume (PV) is recommended. 

For container deployment, in case the MaxMetaspace is not set, in case the native memory balloons, which eventually would lead to the container cgroups's limit breach causing a cgroups OOM-Kill, which is not triggered/controlled by the JVM, but rather by the Red Hat OpenShift Container Platform (RHOCP) node where that container is deployed. For troubleshooting those cases, see the RHOCP node details and the SOS report, in case this was collected. 

Troubleshooting Metaspace

Tools for collaborating that can be used are:

• GC.logs: Metaspace usage/re-caps take a full garbage collection (GC) operation to list those operations on the GC logs.
• VM.info (and or VM.metaspace—see details above): Includes metaspace details (even more so in JDK 17) and shared libraries. To collect VM.info see this solution and to interpret its contents see this solution. 
• Usage of benchmarks: Benchmarks come in handy if one can associate/correlate each usage with a specific version/change in the application, as highlighted in the article How to use Java container awareness in OpenShift 4.
• Heap dump: Heap dump can be used for heap investigations and, although native allocation is outside the heap, some hints can be found inside the heap via Querying Heap Objects (OQL), such as Native Buffer allocation used by Red Hat JBoss Enterprise Application Platform (JBoss EAP) for example. However, VM.info (with Native Tracker) can be more useful depending on the situation. 

Conclusion

JDK 8+ feature in Metaspace is a useful part of the native region and provides a specific region for the allocation of metadata for class loading inside the memory. Even more so, it is not controlled by the Xmx/Xms settings, so it will use the native region and its behavior differs from the previous versions.

Specifically about Metaspace/MaxMetaspace there are pros and cons to setting those values.

• Not setting a hard cap on Metaspace  (i.e., not setting MaxMetaspace), Metaspace may continue to grow. As a worst case, the unbounded Metaspace growth may result in a process death triggered by memory exhaustion.
• Whereas setting a hard cap on Metaspace will incur in OOME if the value is not enough. If setting MaxMetaspace, Metaspace growth is bounded and if that limit is reached by the application use, then the worst-case scenario here is OutOfMemoryErrors in the Java application along with likely high CPU with recurring Full GCs.

One approach that can be considered would be defensive programming, where if anticipating a possible problem in case of a Metaspace leak setting MaxMetaspace JVM flag regardless of the situation to cause an OutOfMemoryException event. Later either save the heap in case it is useful (for a later investigation) or restart the pod immediately viaExitOnOutOfMemoryError.

For container usage, the heap allocation is deduced from MaxRAMPercentage, which already comes by default in OpenJDK images at 50% heap (or 80% heap), so there is no need to set Xmx, Xms, or even MaxRAM directly. The native region will be deduced as the remainder of the container memory. Additionally, I would suggest setting Metaspace/MaxMetaspace it after benchmarking the application. And if the application is not benchmarked I would suggest still setting it.

Finally, benchmarking the application in scenarios such as low, medium, and high request/usage is a great tool, not just for native (off-heap) investigations but heap and container memory as well. It never hurts to log and establish guidelines for the application's CPU and memory usage. Tools such as vm.info and GC logs can also be useful not just during abnormalities but also to establish baselines.

Additional resources

To learn more, read Using Java Container awareness  and Java 17: What’s new in OpenJDK's container awareness.

For any other specific inquiries, please open a case with Red Hat support. Our global team of experts can help you with any issues.

Special thanks to Leticia and Aaron Ogburn for their contributions throughout the last 6y+ years working with Java issues.