Configuring the MongoDB WiredTiger memory cache for RHMAP

This article describes how to configure MongoDB's WiredTiger memory cache in Red Hat Mobile Application Platform (RHMAP) to prevent high-usage memory issues and Nagios alerts. If the WiredTiger cache consumes all the memory available for a container, memory issues and Nagios alerts will occur.

The WiredTiger storage engine is the default storage engine starting in MongoDB version 3.2. It uses MultiVersion Concurrency Control (MVCC) architecture for write operations in order to allow multiple different modifications to the same document at the same time.

WiredTiger also caches data and creates checkpoints to give you the ability to recover anytime it’s necessary. For example, if a MongoDB image deployed in a container fails, it is useful to recover the data that was not persisted. Additionally, WiredTiger can recover un-checkpointed data with its journal files. See the journal documentation and snapshots and checkpoint documentation for more information.

Configuring memory usage

To prevent memory issues and Nagios alerts, the property storage.wiredTiger.engineConfig.cacheSizeGB should be set to a value less than the amount of RAM available in the container. Another option to achieve the same result is through the –wiredTigerCacheSizeGB parameter.

In version 3.2 of MongoDB, the default configuration is to use 1GB of memory or 60% of the available amount of memory when the available amount is larger than one 1GB. This usage percentage was replaced by 50% in version 3.4 of MongoDB. For more details, see the storage.wiredTiger.engineConfig.cacheSizeGB API documentation.

Due to a bug in MongoDB 3.2, a workaround is required to accept values lower than 1Gi. The property configString: cache_size=<cache-size> needs to be set in the MongoDB configuration. The following is an example of setting the property to 600MB:

# storage options - How and where to store data
storage:
    # Directory for datafiles (defaults to /data/db/)
    dbPath: ${MONGODB_DATADIR}
    wiredTiger:
       engineConfig:
           configString: cache_size=600M

Note: The bug "WiredTiger cache size is only configurable in whole gigabytes" can be viewed here.

Avoiding high-usage memory alerts and issues related to MongoDB in RHMAP

In version 4.6.5 or above of RHMAP, the WiredTiger memory cache will be already configured to use 60%/600MB of the default amount of available memory, which is 1GB in the newer RHMAP MongoDB image released. To check this image, see its catalog here. However, in the previous image, memory issues and high-usage memory alerts may be triggered in Nagios because the cache size configuration was not set.

It is recommended that the latest version of the product be used in order to achieve the best user experience and avoid technical debts. If this is not possible, then it is recommended to update all MongoDB instances' configuration to include a proper value for the WiredTiger cache size in both projects: Core and MBaaS.

The following the steps are examples suggested by Shannon Poole, an RHMAP senior software engineer, to perform this configuration manually.

1. Create a ConfigMap, as shown in the following example:

kind: ConfigMap
apiVersion: v1
metadata:
  name: wired-tiger-config
  namespace: rhmap-3-node-mbaas
data:
  mongod.conf: |
    ##
    ## For list of options visit:
    ## https://docs.mongodb.org/manual/reference/configuration-options/
    ##

    # systemLog options - How to do logging
    systemLog:
      # Runs the mongod in a quiet mode that attempts to limit the 
      # amount of output
      quiet: true


    # net options - Network interfaces settings
    net:
      # Specify port number (27017 by default)
      port: 27017


    # storage options - How and where to store data
    storage:
      # Directory for datafiles (defaults to /data/db/)
      dbPath: /var/lib/mongodb/data
      wiredTiger:
        engineConfig:
          configString: cache_size=400M

    # replication options - Configures replication
    replication:
      # Specifies a maximum size in megabytes for the replication 
      # operation log (i.e. the oplog,
      # 5% of disk space by default)
      oplogSizeMB: 64

2. Add the ConfigMap to the deployment configuration (dc) of the MongoDB pod in order to mount it in the right place (/etc/mongod.conf). To edit the deployment configuration, use the following command or the Red hat OpenShift console.

$ oc edit dc/<deployment_config>

Note: See the Consuming in Volumes section of the Red Hat OpenShift documentation for further information.

3. Update the MongoDB replica set container specification, as shown in the following example:

containers:
- volumeMounts:
  - name: wired-tiger-config
    mountPath: /etc/mongod.conf
    subPath: mongod.conf
volumes:
- name: wired-tiger-config
  configMap:
    name: wired-tiger-config

4. Check the pod logs to verify the changes. See if the custom cache size will be there after MongoDB is redeployed. This customization will be described in the logs when MongoDB initializes, as shown in the following example:

=> [Mon Aug 20 13:42:22] wiredTiger cacheSizeGB set to 1
=> [Mon Aug 20 13:42:22] Waiting for local MongoDB to accept connections  …
2018-08-20T13:42:22.837+0000 I STORAGE  [main] Engine custom option: cache_size=600M

Conclusion

If the memory amount allocated for the MongoDB instances will be changed, then it is recommended that you review the configuration and set an adequate value for the WiredTiger cache. There's no need to perform this configuration when the memory resource is bigger than 1GB, since the default set up is 60% of the total available in this situation. However, if a change is done in order to allocate 1GB or less, the recommendation is to customize this value, as explained in this post.