Article

Shaping the Performance of a Linux Distro: Inside Red Hat Enterprise Linux 7

March 17, 2015
Linux
Jeremy Eder

 

Red_Hat_RGB_150px

Backstory

Red Hat's Performance Engineering team is responsible for the performance of many of Red Hat's products.  We cover existing

products such as RHEL, OpenStack Platform, OpenShift and RHEV, as well as newer products like Ceph and CloudForms.

Although these days we contribute extensively to Red Hat's cloud offerings, Red Hat Enterprise Linux remains a core responsibility as the building block for our large ecosystem of customers and partners, plus much of Red Hat's growing product portfolio.

Smoketest Surprise

Some our earliest work on what would become RHEL7 was done on 3.5-ish kernels (somewhere in the Fedora 17 timeframe).  As more and more results started to roll in, we became a bit concerned about the performance of some aspects of the upstream kernel.

While some results were on-par with RHEL6, many were not.  In fact, some OLTP workloads regressed by 20+%, and some latency benchmarks were off double-digits, too!  It was clear that we had our work cut out for us.  And that leads us to...

How we spent 2012 and 2013

The years 2012 and 2013 were spent in the lab making nerdy signs...

Notre Dame Sign

...scoping and characterizing the earliest builds of what would become RHEL7.

Fun stuff like crushing (and crashing!) upstream kernels, and working with partners and customers to ensure a successful release of the newest version of Red Hat Enterprise Linux, version 7.0.  As a company, easily our most important release because not only must it delight our existing RHEL customer base, but it must also become the foundation of our cloud portfolio.  We had to get it right.

As some of the initial evaluators of upstream or test code, our most significant (mostly hidden) contribution to the open source community is providing early adopter feedback to our development teams during the design phase, when code is most flexible.  We need to know things like:

  • How it’s designed.
  • Where are the long poles, and can they be mitigated?
  • What makes sense for defaults?
  • Do we need tracepoints — how do I observe the critical sections under load?
  • Finally, a personal goal…How do we avoid surprising the sysadmins, who are again further downstream, and the very lifeblood of a infrastructure provider like Red Hat.

One of the key discoveries we made as a team throughout a decade of supporting customers on RHEL, was an obsession the prioritization of customer experience.  We knew we wanted our customers to be absolutely thrilled with by the stability, reliability, security and most importantly ;-) the performance of Red Hat Enterprise Linux 7.

To the way-back machine...

We'd actually begun the journey towards workload-specific tuning in RHEL5 with the introduction of ktune.  ktune provides some nascent tuning profiles for a very small set of workloads.  We did not expand much on ktune in the RHEL5 product.

evolution

With the GA of RHEL6.0, we introduced the tuned package.  I like to describe tuned as a "tuning profile delivery mechanism", and ends up being our group's primary feedback loop into the Red Hat's product line.  If you haven't heard of tuned in RHEL6, trying it is as simple as:

# yum install tuned
# service tuned start
# tuned-adm list
# tuned-adm profile throughput-performance

We received lots of great feedback from our partners and customers about performance in RHEL6.  Coupled with additional R&D, this feedback allowed us to confidently expand the reach of tuned and it's profiles beyond RHEL.  So in addition to the profiles shipped with RHEL, Red Hat began to ship profiles for a growing list of products such as RHEV, RHEL OpenStack Platform, OpenShift, RHEL Atomic and Red Hat Storage.

Here's a chart depicting the tuned profile inheritance feature in RHEL7.  RHEL includes 3 "parent" profiles, a handful of child profiles, and even some grand-children.  Users are free to customize existing profiles, create their own, or use none at all.

tuned-inheritance

Based on extensive testing, these profiles typically boost performance in the double-digit percent range "for free".  Just some of the benefits of Red Hat's Enterprise-hardened distribution versus free/community software.

Time to make the donuts RHEL7

Through 2+ years and 4+ minor releases of RHEL6, our team gained valuable experience with the tuning profiles customers began adopting.  As we began working on RHEL7, the next logical step to improve out-of-the-box performance for our customers, was to meticulously validate upstream Linux kernel defaults through the lens of enterprise datacenter and cloud environments.

Although it's impossible to identify one specific set of tunings that help all workloads, this effort led us to a set of changes to default kernel compile-time settings and sysctl tunings that would boost performance of most workloads well passed upstream defaults.  These changes were so impactful, that we collaborated with our kernel engineering teams to further validate, and propose tuned be enabled by default in RHEL7.  After many months of discussion and testing, this proposal was accepted and delivered in the RHEL7.0 GA.

Note:  as a further optimization, tuned will automatically customize itself based on what version of RHEL (Workstation/Server) is installed, and whether it's running on virtualization or bare metal.

Summary

Prior to beginning efforts on RHEL7 in earnest, Red Hat's Platform Business Unit continued to deliver minor updates to RHEL5 and RHEL6 simultaneously, along with extending support to 10 years.  Concurrent delivery of 3 major streams of RHEL is only possible because of the depth of talent (and frankly, depth of character) present in our engineering teams.

As a fairly large team of performance engineers, we remain focused on ensuring RHEL continues to be the trusted, rock-solid foundation upon which customers and partners should remain confident to run their business.

Last updated: February 26, 2024