Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Shaping the Performance of a Linux Distro: Inside Red Hat Enterprise Linux 7

March 17, 2015
Jeremy Eder
Related topics:
Linux
Related products:
Red Hat Enterprise Linux

Share:

     

    Red_Hat_RGB_150px

    Backstory

    Red Hat's Performance Engineering team is responsible for the performance of many of Red Hat's products.  We cover existing

    products such as RHEL, OpenStack Platform, OpenShift and RHEV, as well as newer products like Ceph and CloudForms.

    Although these days we contribute extensively to Red Hat's cloud offerings, Red Hat Enterprise Linux remains a core responsibility as the building block for our large ecosystem of customers and partners, plus much of Red Hat's growing product portfolio.

    Smoketest Surprise

    Some our earliest work on what would become RHEL7 was done on 3.5-ish kernels (somewhere in the Fedora 17 timeframe).  As more and more results started to roll in, we became a bit concerned about the performance of some aspects of the upstream kernel.

    While some results were on-par with RHEL6, many were not.  In fact, some OLTP workloads regressed by 20+%, and some latency benchmarks were off double-digits, too!  It was clear that we had our work cut out for us.  And that leads us to...

    How we spent 2012 and 2013

    The years 2012 and 2013 were spent in the lab making nerdy signs...

    Notre Dame Sign

    ...scoping and characterizing the earliest builds of what would become RHEL7.

    Fun stuff like crushing (and crashing!) upstream kernels, and working with partners and customers to ensure a successful release of the newest version of Red Hat Enterprise Linux, version 7.0.  As a company, easily our most important release because not only must it delight our existing RHEL customer base, but it must also become the foundation of our cloud portfolio.  We had to get it right.

    As some of the initial evaluators of upstream or test code, our most significant (mostly hidden) contribution to the open source community is providing early adopter feedback to our development teams during the design phase, when code is most flexible.  We need to know things like:

    • How it’s designed.
    • Where are the long poles, and can they be mitigated?
    • What makes sense for defaults?
    • Do we need tracepoints — how do I observe the critical sections under load?
    • Finally, a personal goal…How do we avoid surprising the sysadmins, who are again further downstream, and the very lifeblood of a infrastructure provider like Red Hat.

    One of the key discoveries we made as a team throughout a decade of supporting customers on RHEL, was an obsession the prioritization of customer experience.  We knew we wanted our customers to be absolutely thrilled with by the stability, reliability, security and most importantly ;-) the performance of Red Hat Enterprise Linux 7.

    To the way-back machine...

    We'd actually begun the journey towards workload-specific tuning in RHEL5 with the introduction of ktune.  ktune provides some nascent tuning profiles for a very small set of workloads.  We did not expand much on ktune in the RHEL5 product.

    evolution

    With the GA of RHEL6.0, we introduced the tuned package.  I like to describe tuned as a "tuning profile delivery mechanism", and ends up being our group's primary feedback loop into the Red Hat's product line.  If you haven't heard of tuned in RHEL6, trying it is as simple as:

    # yum install tuned
    # service tuned start
    # tuned-adm list
    # tuned-adm profile throughput-performance

    We received lots of great feedback from our partners and customers about performance in RHEL6.  Coupled with additional R&D, this feedback allowed us to confidently expand the reach of tuned and it's profiles beyond RHEL.  So in addition to the profiles shipped with RHEL, Red Hat began to ship profiles for a growing list of products such as RHEV, RHEL OpenStack Platform, OpenShift, RHEL Atomic and Red Hat Storage.

    Here's a chart depicting the tuned profile inheritance feature in RHEL7.  RHEL includes 3 "parent" profiles, a handful of child profiles, and even some grand-children.  Users are free to customize existing profiles, create their own, or use none at all.

    tuned-inheritance

    Based on extensive testing, these profiles typically boost performance in the double-digit percent range "for free".  Just some of the benefits of Red Hat's Enterprise-hardened distribution versus free/community software.

    Time to make the donuts RHEL7

    Through 2+ years and 4+ minor releases of RHEL6, our team gained valuable experience with the tuning profiles customers began adopting.  As we began working on RHEL7, the next logical step to improve out-of-the-box performance for our customers, was to meticulously validate upstream Linux kernel defaults through the lens of enterprise datacenter and cloud environments.

    Although it's impossible to identify one specific set of tunings that help all workloads, this effort led us to a set of changes to default kernel compile-time settings and sysctl tunings that would boost performance of most workloads well passed upstream defaults.  These changes were so impactful, that we collaborated with our kernel engineering teams to further validate, and propose tuned be enabled by default in RHEL7.  After many months of discussion and testing, this proposal was accepted and delivered in the RHEL7.0 GA.

    Note:  as a further optimization, tuned will automatically customize itself based on what version of RHEL (Workstation/Server) is installed, and whether it's running on virtualization or bare metal.

    Summary

    Prior to beginning efforts on RHEL7 in earnest, Red Hat's Platform Business Unit continued to deliver minor updates to RHEL5 and RHEL6 simultaneously, along with extending support to 10 years.  Concurrent delivery of 3 major streams of RHEL is only possible because of the depth of talent (and frankly, depth of character) present in our engineering teams.

    As a fairly large team of performance engineers, we remain focused on ensuring RHEL continues to be the trusted, rock-solid foundation upon which customers and partners should remain confident to run their business.

    Last updated: February 26, 2024

    Recent Posts

    • How to build a Model-as-a-Service platform

    • How Quarkus works with OpenTelemetry on OpenShift

    • Our top 10 articles of 2025 (so far)

    • The benefits of auto-merging GitHub and GitLab repositories

    • Supercharging AI isolation: microVMs with RamaLama & libkrun

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue