Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Metaspace setting and tuning in JDK 8+ Applications (in and outside containers)

July 19, 2024
Francisco De Melo Junior
Related topics:
JavaJava MicroservicesKubernetesRuntimes
Related products:
Red Hat build of OpenJDKRed Hat build of QuarkusRed Hat Data Grid

Share:

    Since JDK 8, including upper versions, Metaspace has been set apart from the main heap section of memory and defined via MetaspaceSize and MaxMetaspace flags. Objectively, this means metadata of loading classes in Java is located in the non-heap region also known as the native region of the memory (again, not inside the heap).

    This Metaspace region is a change from previous versions, which had Java Metaspace as part of the heap in the so-called "Permanent Generation" and was set using the Java Virtual Machine (JVM) flag: PermSize and MaxPermSize. 

    Although this discussion is very well settled, this article aims to review Metaspace, its tuning, and the consequences for container and non-container usage. It also adds troubleshooting steps in case those are needed.

    This article complements the previous this other one: How to use Java container awareness in OpenShift 4 providing details for the off-heap section Metaspace. This is a brief introduction to this subject, not a deep dive.

    Metaspace explanation

    As briefly mentioned above, Metaspace was introduced on JDK 8+, therefore, in the case of OutOfMemory Exception, changed from the  Java.Lang.OutOfMemoryError: PermGen to OutOfMemoryError: Metaspace. Consequently, this is the first indication that the Java version is being used above JDK 7.

    Metaspace does not occupy the heap region but rather the native space of the JVM, so it is not bounded by the Xmx setting.

    Metaspace will allocate regions of memory in the so-called metaspace chunks, or metachunks inside the Virtual Spaces, which are areas of contiguous address space provided by the OS. This allocation is done on demand. More information in this solution: How does the JVM divide the Metaspace in the memory?

    Also in that solution, it is explained that in JDK 11, the VM.metaspace option was introduced in Java diagnostics, which provides complete detailed information about Metaspace, including the chunks, the size, the InitialBootClassLoaderMetaspaceSize, UseCompressedClassPointers, and CompressedClassSpaceSize. Also, detailed information on the chunks:

    $JDK11_HOME/bin/jcmd 27689 VM.metaspace
    27689:
    Total Usage ( 1490 loaders): <--- number of loaders
     Non-Class: 4312 chunks,     98.45 MB capacity,    91.95 MB ( 93%) used,     6.23 MB (  6%) free,     4.60 KB ( <1%) waste,   269.50 KB ( <1%) overhead, deallocated: 28367 blocks with 4.26 MB
         Class: 2402 chunks,     17.00 MB capacity,    13.64 MB ( 80%) used,     3.21 MB ( 19%) free,   328 bytes ( <1%) waste,   150.12 KB ( <1%) overhead, deallocated: 1753 blocks with 919.75 KB
          Both: 6714 chunks,    115.45 MB capacity,   105.60 MB ( 91%) used,     9.44 MB (  8%) free,     4.92 KB ( <1%) waste,   419.62 KB ( <1%) overhead, deallocated: 30120 blocks with 5.16 MB

    Using Java Diagnostic in JDK 17 brings GC details/information about the metaspace (which on JDK 17 has a buddy algorithm) and brings more details on the Metaspace Reclaim Policy:

    $ jcmd PID VM.info --->
    ### Metaspace
    MaxMetaspaceSize: 96.00 MB <----------------------------- Max Metaspace size
    CompressedClassSpaceSize: 80.00 MB <--------------------- compressed class space size == CCSS
    Initial GC threshold: 32.00 MB <------------------------- initial GC threshold
    Current GC threshold: 65.25 MB  <------------------------ current GC threshold
    CDS: on <------------------------------------------------ Class Data Sharing 
    MetaspaceReclaimPolicy: balanced <----------------------- Metaspace Reclaim Policy: balanced
    - commit_granule_bytes: 65536.
    - commit_granule_words: 8192.
    - virtual_space_node_default_size: 1048576.
    - enlarge_chunks_in_place: 1.
    - new_chunks_are_fully_committed: 0.
    - uncommit_free_chunks: 1.
    - use_allocation_guard: 0.
    - handle_deallocations: 1.

    Below OpenJDK 17 VM.metaspace:

    sh-5.1$ jcmd 150 VM.metaspace
    150:
    Total Usage - 432 loaders, 12819 classes (1420 shared):
      Non-Class: 2103 chunks,     51.69 MB capacity,   51.31 MB (>99%) committed,    50.98 MB ( 99%) used,   338.89 KB ( <1%) free,     2.15 KB ( <1%) waste , deallocated: 87 blocks with 14.46 KB
          Class:  818 chunks,      7.62 MB capacity,    7.55 MB (>99%) committed,     7.32 MB ( 96%) used,   234.30 KB (  3%) free,   152 bytes ( <1%) waste , deallocated: 348 blocks with 84.27 KB
           Both: 2921 chunks,     59.30 MB capacity,   58.87 MB (>99%) committed,    58.30 MB ( 98%) used,   573.19 KB ( <1%) free,     2.30 KB ( <1%) waste , deallocated: 435 blocks with 98.73 KB
    
    Virtual space:
      Non-class space:       64.00 MB reserved,      51.31 MB ( 80%) committed,  1 nodes.
          Class space:       80.00 MB reserved,       7.56 MB (  9%) committed,  1 nodes.
                 Both:      144.00 MB reserved,      58.88 MB ( 41%) committed. 
    Chunk freelists:
       Non-Class:
     16m: (none)
      8m:    2, capacity=16.00 MB, committed=0 bytes (  0%)
      4m:    2, capacity=8.00 MB, committed=0 bytes (  0%)
      2m: (none)
    ...
    Total word size: 33.01 MB, committed: 0 bytes (  0%)
    Waste (unused committed space):(percentages refer to total committed size 58.88 MB):
            Waste in chunks in use:      2.30 KB ( <1%)
            Free in chunks in use:    573.19 KB ( <1%)
                    In free chunks:      0 bytes (  0%)
    Deallocated from chunks in use:     98.73 KB ( <1%) (435 blocks)
                           -total-:    674.22 KB (  1%)
    
    chunk header pool: 2930 items, 207.36 KB.
    Internal statistics:
    
    num_allocs_failed_limit: 6.
    num_arena_births: 866.
    num_arena_deaths: 2.
    num_vsnodes_births: 2.
    num_vsnodes_deaths: 0.
    num_space_committed: 942.
    num_space_uncommitted: 0.
    num_chunks_returned_to_freelist: 8.
    num_chunks_taken_from_freelist: 2924.
    num_chunk_merges: 6.
    num_chunk_splits: 2075.
    num_chunks_enlarged: 1621.
    num_inconsistent_stats: 0.
    
    Settings: <--- will be also in VM.info
    MaxMetaspaceSize: 96.00 MB
    CompressedClassSpaceSize: 80.00 MB
    Initial GC threshold: 32.00 MB
    Current GC threshold: 89.00 MB
    CDS: on
     - commit_granule_bytes: 65536.
     - commit_granule_words: 8192.
     - virtual_space_node_default_size: 8388608.
     - enlarge_chunks_in_place: 1.
     - use_allocation_guard: 0.

    Details on how to set adequate Metaspace and MaxMetaspace

    A few core points below:

    • The JVM flag: Metaspace does not set the minimal size of the Metaspace, but rather it sets the initial size of the Metaspace segment. Evidently, there is no minimal usage for metaspace given the metadata will be loaded dynamically accordingly to the classes are loaded. 
    • MaxMetaspace sets the maximum size of the Metaspace, as in the upper boundary. Crossing that boundary leads to the OutOfMemoryError: Metaspace (abbreviated as OOME Metaspace).
    • Metaspace is not the only component for native usage, 

    Setting initial and max sizes inside containers

    The question from the explanation above is: should I set the initial and max sizes? 

    In terms of usage, each application will load a certain amount of classes so the use is on a case-by-case basis, however, usually even large deployments likely will use less than 1GB or 2GB of Metaspace, which would be about 1 to two billion bytes of data.

    It depends on the application and the response to the OOME that crossing the max limit will cause. Let's see two scenarios for discussion below.

    The scenarios will be as follows:

    • Scenario 1: The application has been developed in JDK 8+ and benchmarked.
    • Scenario 2: The application has been migrated recently and has not been benchmarked.
    • Scenario 3: The application has been developed for container and benchmarked for its usage.
    • Scenario 4: The application has not been developed for container and no consumption table is available.

    Scenario 1: The application has been developed in JDK 8+ and benchmarked

    In case the application was developed for JDK 8+ and metaspace was already taken into consideration in its development phase is likely the memory consumption benchmarks already take into consideration the adequate initial and max metaspace size. It might be useful not to have max sizes given the already expected allocation. Otherwise, it can be useful to set the max size to trigger an OOME: Metaspace in case of abnormality.

    Scenario 2: The application has been migrated recently and has not been benchmarked

    In case the application was not developed for JDK 8+ but instead was migrated, it is possible the metaspace usage was not necessarily benchmarked after its migration phase. It might be useful for the user to set initial settings and max settings to avoid unexpected usage, therefore triggering an OOME: Metaspace in case of abnormality (several OutOfMemory Exceptions can happen, Metaspace is just one of them).

    As a corollary from the above, should I set the initial and max size inside a container? Let's see two scenarios for discussion below.

    Scenario 3: The application has been developed for container and benchmarked for its usage

    In case the application was developed for containers and metaspace was already taken into consideration in the container size, as in benchmarked already taken into consideration the adequate initial and max metaspace size, it might be useful not to have max sizes given the already expected allocation. 

    Otherwise, it can be useful to set the max size to trigger an OOME: Metaspace in case of abnormality, the OOME needs to be properly handled for the information to be saved for posterior investigation because the container restart would clean any non-persisted file, being heap dump, crash file, thread dump, or VM.info.  For container deployment, in case the MaxMetaspace is set, in case the native memory balloons an OOME will be triggered and this will cause the exit of the JVM and without process, the container will exit, and subsequently the pod will restart. 

    Scenario 4: The application has not been developed for container and no consumption table is available

    In case the application was not necessarily developed for containers and therefore memory consumption inside didn't already take into consideration the container size, it might be adequate to set initial and max metaspace sizes. Regarding the specific value to set, it can be small to cause OOME sooner or a higher value to avoid OOME conditions. 

    Either way, given an unfamiliarity with the memory usage of the application inside the container, it can be useful to set memory boundaries in the container and set initial and max sizes to trigger OOME: Metaspace in case of abnormality. Again, the OOME needs to be properly handled for the information to be saved for posterior investigation because the container restart would clean any non-persisted file, being heap dump, crash file, thread dump, or VM.info. 

    In this matter, several containers are deployed using ExitOnOutOfMemoryError JVM flag, such as JBoss EAP 7, which is not an issue, but the container will exit on the spot (usually with code 3 on the pod log) and there will be no heap saved. Depending on the scenario, in case the heap investigation is required, usage of CrashOnOutOfMemoryError over ExitOnOutOfMemoryError or sending the heap for a Persistence Volume (PV) is recommended. 

    For container deployment, in case the MaxMetaspace is not set, in case the native memory balloons, which eventually would lead to the container cgroups's limit breach causing a cgroups OOM-Kill, which is not triggered/controlled by the JVM, but rather by the Red Hat OpenShift Container Platform (RHOCP) node where that container is deployed. For troubleshooting those cases, see the RHOCP node details and the SOS report, in case this was collected. 

    Troubleshooting Metaspace

    Tools for collaborating that can be used are:

    • GC.logs: Metaspace usage/re-caps take a full garbage collection (GC) operation to list those operations on the GC logs.
    • VM.info (and or VM.metaspace—see details above): Includes metaspace details (even more so in JDK 17) and shared libraries. To collect VM.info see this solution and to interpret its contents see this solution. 
    • Usage of benchmarks: Benchmarks come in handy if one can associate/correlate each usage with a specific version/change in the application, as highlighted in the article How to use Java container awareness in OpenShift 4.
    • Heap dump: Heap dump can be used for heap investigations and, although native allocation is outside the heap, some hints can be found inside the heap via Querying Heap Objects (OQL), such as Native Buffer allocation used by Red Hat JBoss Enterprise Application Platform (JBoss EAP) for example. However, VM.info (with Native Tracker) can be more useful depending on the situation. 

    Conclusion

    JDK 8+ feature in Metaspace is a useful part of the native region and provides a specific region for the allocation of metadata for class loading inside the memory. Even more so, it is not controlled by the Xmx/Xms settings, so it will use the native region and its behavior differs from the previous versions.

    Specifically about Metaspace/MaxMetaspace there are pros and cons to setting those values.

    • Not setting a hard cap on Metaspace  (i.e., not setting MaxMetaspace), Metaspace may continue to grow. As a worst case, the unbounded Metaspace growth may result in a process death triggered by memory exhaustion.
    • Whereas setting a hard cap on Metaspace will incur in OOME if the value is not enough. If setting MaxMetaspace, Metaspace growth is bounded and if that limit is reached by the application use, then the worst-case scenario here is OutOfMemoryErrors in the Java application along with likely high CPU with recurring Full GCs.

    One approach that can be considered would be defensive programming, where if anticipating a possible problem in case of a Metaspace leak setting MaxMetaspace JVM flag regardless of the situation to cause an OutOfMemoryException event. Later either save the heap in case it is useful (for a later investigation) or restart the pod immediately viaExitOnOutOfMemoryError.

    For container usage, the heap allocation is deduced from MaxRAMPercentage, which already comes by default in OpenJDK images at 50% heap (or 80% heap), so there is no need to set Xmx, Xms, or even MaxRAM directly. The native region will be deduced as the remainder of the container memory. Additionally, I would suggest setting Metaspace/MaxMetaspace it after benchmarking the application. And if the application is not benchmarked I would suggest still setting it.

    Finally, benchmarking the application in scenarios such as low, medium, and high request/usage is a great tool, not just for native (off-heap) investigations but heap and container memory as well. It never hurts to log and establish guidelines for the application's CPU and memory usage. Tools such as vm.info and GC logs can also be useful not just during abnormalities but also to establish baselines.

    Additional resources

    To learn more, read Using Java Container awareness  and Java 17: What’s new in OpenJDK's container awareness.

    For any other specific inquiries, please open a case with Red Hat support. Our global team of experts can help you with any issues.

    Special thanks to Leticia and Aaron Ogburn for their contributions throughout the last 6y+ years working with Java issues.

    Last updated: January 15, 2025

    Related Posts

    • OpenJDK 8 and 11: Still in safe hands

    • Using Red Hat Application Migration Toolkit to see the impact of migrating to OpenJDK

    • Securely connect Quarkus and Red Hat Data Grid on Red Hat OpenShift

    • Not all OpenJDK 12 builds include Shenandoah: Here's why

    Recent Posts

    • Storage considerations for OpenShift Virtualization

    • Upgrade from OpenShift Service Mesh 2.6 to 3.0 with Kiali

    • EE Builder with Ansible Automation Platform on OpenShift

    • How to debug confidential containers securely

    • Announcing self-service access to Red Hat Enterprise Linux for Business Developers

    What’s up next?

    Read Writing a Kubernetes Operator in Java Cheat Sheet to learn how to create a Kubernetes Operator in Java using Quarkus.

    Get the cheat sheet
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue