Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Controlling resources with cgroups for performance testing

September 21, 2015
Frédéric Giloux
Related topics:
Linux
Related products:
Red Hat Enterprise Linux

Share:

    Introduction

    Today I want to write about the options available to limit resources in use for running performance tests in a shared environment. A very powerful tool for this is cgroups [1] - a Linux kernel feature that allows limiting the resource usage (CPU, memory, disk I/O, etc..) of a collection of processes.
    Nowadays it is easy with virtual machines or container technologies, like Docker, which is using cgroups by the way, to compartmentilize applications and make sure that they are not eating resources that have not been allocated to them. But you may face, as I did, situations where these technologies are not available and if your application happens to run on a Linux distribution like RHEL 6 and later you can directly rely on cgroups for this purpose. 

    I was visiting a customer who was in the process of migrating his servers to a virtualisation platform. At that point in time they had a virtualised production platform, an integration platform where the tests were to be performed, was based on physical hardware having specs, which were not aligned with the production platform the application was running on. A server of the integration platform was indeed used to run two applications, split into two different VMs in production. But I wasn't either able or willing to create load for both applications. The performance tests were designed for only one of them. If I was to run them just like that I would not have results representative of the production platform. The customer was indeed putting the virtualised environment into question as they were seeing degraded performance compared to the physical environment used for the integration tests. I decided to use Linux capabilities to limit CPU access and make my tests an acceptable simulation of the production. By doing this I helped the customer regaining trust in their new virtualisation environment.

    Another scenario, where one may use cgroups is for evaluating the resource specifications required by an application. It is then easy on the same server to perform tests with one, two, four or sixteen CPUs (provided you have them available). You could tell me that the same can easily be achieved with the hypervisor of a virtualised environment and you would be right. But if you don't have access to it? The process for getting the person having the rights to apply your changes may take days rather than minutes.

    So let's come to the point now and look at how easy it is to configure cgroups for this purpose after I have enumerated a few important caveats:

    • You need sudo rights to change the cgroups configuration
    • Be careful if you already have cgroups configured on your system. It is easy to destroy your existing configuration. Don't blame me for that. Know what you are doing.

    If one of these points prevent you of using cgroups, taskset described later in this blog entry may be a good alternative. Though, it only applies to CPU affinity whereas cgroups can be used to limit other resource types as well.

    Installation

    Cgroup is already installed on RHEL 7.

    On RHEL 6 the libcgroup package is the easiest way to work with cgroups and its installation is straightforward: yum install libcgroup

    However this package is deprecated on RHEL 7 since it can easily create conflicts with the default cgroup hierarchy.

    Configuration

    Cgroups are organized hierarchically, like processes, and child cgroups inherit some of the attributes of their parents. A subsystem (also called resource controller) represents a single resource, such as CPU time or memory. Note that the configuration differs between RHEL 6 and 7. See Red Hat Enterprise Linux 6/7 Resource Management Guide [2] and [2'] for more information.

    RHEL 7

    By default systemd automatically creates a hierarchy of slice, scope and service units. Also systemd automatically mounts hierarchies for important resource controllers in the /sys/fs/cgroup/ directory.

    Services are started by Systemd whereas scope groups are externally created processes. Slices organize a hierarchy in which scopes and services are placed. Processes are attached to services and scopes, not to slices. A service is a process or a group of processes, which are started by systemd, whereas a scope is a process or a group of processes, which are created externally (fork).

    So let's start a new process. The command systemd-run allows to start a new process and to attach it to a specific transient unit. If the unit does not exist it will get created. In the same way a slice can be created with the option --slice. Per default the command creates a new service. The option --scope allows the creation of a scope.

    # sudo systemd-run --unit=fredunit --scope --slice=fredslice top

    We have now a new slice and it is possible to visualise it:

    # systemctl status fredslice.slice
    fredslice.slice
    Loaded: loaded
    Active: active since Sat 2015-08-15 09:04:09 CEST; 9min ago
    CGroup: /fredslice.slice
    └─fredunit.scope
    └─4277 /bin/top

    In the same way it is possible to display the new scope:

    # systemctl status fredunit.scope
    fredunit.scope - /bin/top
    Loaded: loaded (/run/systemd/system/fredunit.scope; static)
    Drop-In: /run/systemd/system/fredunit.scope.d
    └─90-Description.conf, 90-SendSIGHUP.conf, 90-Slice.conf
    Active: active (running) since Sat 2015-08-15 09:04:09 CEST; 8min ago
    CGroup: /fredslice.slice/fredunit.scope
    └─4277 /bin/top

    Transient cgroups are stored in /run/systemd/system.

    Let look at our new process:

    # cat /proc/4277/cgroup
    10:hugetlb:/
    9:perf_event:/
    8:blkio:/
    7:net_cls:/
    6:freezer:/
    5:devices:/
    4:memory:/
    3:cpuacct,cpu:/
    2:cpuset:/
    1:name=systemd:/fredslice.slice/fredunit.scope

    Cgroups use subsystems (also called resource controllers) to represent a single resource, such as CPU time or memory. cpuset [3] provides a mechanism for assigning a set of CPUs and Memory nodes to a set of tasks.

    If the cgroup is not already mounted it can easily be done:

    # sudo mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpuset

    We can now navigate to the cpuset hierarchy and create a new cpuset called fred:

    # cd /sys/fs/cgroup/cpuset
    # sudo mkdir fred

    It is possible to move to the newly created cpuset and to configure it, for instance to allocate the 1st CPU to it

    # cd fred
    # echo 0 | sudo tee -a cpuset.cpus

    We will need a memory node as well

    # echo 0 | sudo tee -a cpuset.mems

    And we can attach our process to the newly created cpuset

    # echo 4277 | sudo tee -a tasks

    Let look at the process configuration again. The cpuset fred is now attached to it:

    # more /proc/4277/cgroup
    10:hugetlb:/
    9:perf_event:/ bin
    8:blkio:/
    7:net_cls:/
    6:freezer:/
    5:devices:/
    4:memory:/
    3:cpuacct,cpu:/
    2:cpuset:/fred
    1:name=systemd:/fredslice.slice/fredunit.scope

    For tests we may also want to limit the amount of memory available to processes. Therefore we can use the memory limit setting:

    # sudo systemctl set-property --runtime fredunit.scope MemoryLimit=1G

    And this now appears at the process level:

    # more /proc/4277/cgroup
    10:hugetlb:/
    9:perf_event:/
    8:blkio:/
    7:net_cls:/
    6:freezer:/
    5:devices:/
    4:memory:/fredslice.slice/fredunit.scope
    3:cpuacct,cpu:/
    2:cpuset:/fred
    1:name=systemd:/fredslice.slice/fredunit.scope

    After reloading the systemd daemon there is now an additional file 90-MemoryLimit.conf in the cgroup:

    # sudo systemctl daemon-reload
    # systemctl status fredunit.scope
    fredunit.scope - /bin/top
    Loaded: loaded
    Drop-In: /run/systemd/system/fredunit.scope.d
    └─90-Description.conf, 90-MemoryLimit.conf, 90-SendSIGHUP.conf, 90-Slice.conf
    Active: active (running) since Sat 2015-08-15 12:07:08 CEST; 19min ago
    CGroup: /fredslice.slice/fredunit.scope
    └─14869 /bin/top

    This file contains the memory limit

    # sudo cat /run/systemd/system/fredunit.scope.d/90-MemoryLimit.conf
    [Scope]
    MemoryLimit=1073741824

    Looks good!

    We have achieved our aim of limiting a process to a single CPU and to 1G of memory.

    Using the exact same mechanisms it is possible to start a shell. The good thing is that every process started from this shell will then inherit the cgroup configuration.

    To start the shell and to set its memory limit:

    # sudo systemd-run --unit=fredunit --scope --slice=fredslice sh
    # sudo systemctl set-property --runtime fredunit.scope MemoryLimit=1G
    # sudo systemctl daemon-reload

    We can control that the shell process has been bound to the cgroup and that the memory limitation has been applied:

    # systemctl status fredunit.scope
    systemctl status fredunit.scope
    fredunit.scope - /bin/sh
    Loaded: loaded
    Drop-In: /run/systemd/system/fredunit.scope.d
    └─90-Description.conf, 90-MemoryLimit.conf, 90-SendSIGHUP.conf, 90-Slice.conf
    Active: active (running) since Sat 2015-08-15 12:37:30 CEST; 3min 0s ago
    CGroup: /fredslice.slice/fredunit.scope
    └─16842 /bin/sh

    Again we add the shell process to the tasks of the fred cpuset we have created earlier and it gets reflected in the cgroup attached to the process:

    # echo 16842 | sudo tee -a tasks
    # more /proc/16842/cgroup
    10:hugetlb:/
    9:perf_event:/
    8:blkio:/
    7:net_cls:/
    6:freezer:/
    5:devices:/
    4:memory:/fredslice.slice/fredunit.scope
    3:cpuacct,cpu:/
    2:cpuset:/fred
    1:name=systemd:/fredslice.slice/fredunit.scope

    From the newly created shell we can start an application:

    # top

    And we can control in another shell that the new process has inherited the cgroup configuration:

    # systemctl status fredunit.scope
    fredunit.scope - /bin/sh
    Loaded: loaded
    Drop-In: /run/systemd/system/fredunit.scope.d
    └─90-Description.conf, 90-MemoryLimit.conf, 90-SendSIGHUP.conf, 90-Slice.conf
    Active: active (running) since Sat 2015-08-15 12:37:30 CEST; 5min ago
    CGroup: /fredslice.slice/fredunit.scope
    ├─16842 /bin/sh
    └─17163 top# cat /proc/17163/cgroup
    10:hugetlb:/
    9:perf_event:/
    8:blkio:/
    7:net_cls:/
    6:freezer:/
    5:devices:/
    4:memory:/fredslice.slice/fredunit.scope
    3:cpuacct,cpu:/
    2:cpuset:/fred
    1:name=systemd:/fredslice.slice/fredunit.scope

    Awesome!

    RHEL 6

    I haven't tested the settings for RHEL 6 but by using libcgroup cgroups can be configured in /etc/cgconfig.conf.

    The default /etc/cgconfig.conf file installed with the libcgroup package creates and mounts an individual hierarchy for each subsystem, and attaches the subsystems to these hierarchies. You can mount subsystems and define groups with different accesses to CPU and memory as follows:

    mount {
    cpuset = /cgroup/cpuset;
    memory = /cgroup/memory;
    }# no limitation group
    group nolimit {
    cpuset {
    # No alternate memory nodes if the system is not NUMA
    cpuset.mems="0";
    # Make my 2 CPU cores available to tasks
    cpuset.cpus="0,1";
    }
    memory {
    # Allocate my 4 GB of memory to tasks
    memory.limit_in_bytes="4G";
    }
    }# group with limitation
    group limited {
    cpuset {
    # No alternate memory nodes if the system is not NUMA
    cpuset.mems="0";
    # Make only one of my CPU cores available to tasks
    cpuset.cpus="0";
    }
    memory {
    # Allocate at most 2 GB of memory to tasks
    memory.limit_in_bytes="2G";
    }
    }

    You must restart the cgconfig service for the changes in /etc/cgconfig.conf to take effect. Note that restarting this service causes the entire cgroup hierarchy to be rebuilt, which removes any previouly existing cgroups.

    To start a process in a control group: cgexec -g controllers:path_to_cgroup command arguments
    It is also possible to add the --sticky option before the command to keep any child processes in the same cgroup.

    Taskset as an alternative

    If you don't have the rights for using cgroups or you only want to limit the number of CPU cores used by your application it is still possible to do it with taskset. You would then rely on the JVM settings for memory limitation if you want to test a java application. This is obviously not the same but it may be an acceptable approximation. As you can read in the "man" description taskset [4] is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs.

    For instance if you have one java process running taskset -p provides information on the current affinity mask:

    # ps -ef | grep java
    frederic 971 1 99 17:56 pts/0 00:00:09 java...
    # taskset -p 971
    pid 971's current affinity mask: 3

    With taskset -cp you can then get the list of processors that your process is bound to:

    # taskset -cp 971
    pid 971's current affinity list: 0,1

    How to interpret the result of these two commands?

    Let look at what "man" says: "The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU."

    I have two CPUs 0x01 and 0x10, which are allocated to my process, which matches the mask 0x11 and this is 3 in base 16.

    Let take another example that may explain it better. If you have 8 processors, they will be numbered this way:
    00000001
    00000010
    00000100
    00001000
    00010000
    00100000
    01000000
    10000000
    if your process has access to the 8 of them, the mask will be 11111111, which is 0x1 + 0x2 + 0x4 + 0x8 + 0x10 + 0x20 + 0x40 + 0x80 = 0xff in base 16.
    if your process has only access to the 1st and 2nd one, as it was the case for me the mask is 11 or 3 in base 16.
    now if the process has only access to the 1st and the 8th one, the mask would be 10000001, which is 0x51 in base 16.

    Note my process runs in a virtual machine, which I have allocated two CPUs to. It is not that I should through my laptop to the garbage. ;o)

    Well, now that we know what we are doing we can allocate the process to a single CPU, the first one for instance.

    taskset -p is the command to use. To have the process bound to the first processor:

    # taskset -p 1 971
    pid 971's current affinity mask: 3
    pid 971's new affinity mask: 1

    if you wanted to run a new command rather than operating on an existing one, which is actually the default behavior it is not more complicate: taskset [arguments]

    Et voilà!

    This blog post was originally published at: https://bricks4bridges.wordpress.com/2015/08/16/cgroups-for-perf-tests/

    [1] https://en.wikipedia.org/wiki/Cgroups
    [2] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu_and_memory-use_case.html
    [2'] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/pdf/Resource_Management_Guide/Red_Hat_Enterprise_Linux-7-Resource_Management_Guide-en-US.pdf
    [3] https://www.kernel.org/doc/Documentation/cgroups/cpusets.txt
    [4] https://en.wikipedia.org/wiki/Processor_affinity

    PS: I hurried up for writing this blog entry as it may get quickly deprecated with the rocket adoption of containers ;-)

    Last updated: February 23, 2024

    Recent Posts

    • Container starting and termination order in a pod

    • More Essential AI tutorials for Node.js Developers

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue