Low Latency Performance Tuning for Red Hat Enterprise Linux 7

Counting micro-nanoseconds?  We are, because we know our customers are.  Some of the world's largest stock exchanges including the Chicago Mercantile Exchange (CME), New York Stock Exchange (NYSE), E*TRADE, Union Bank, countless hedge funds and high-frequency trading shops run on Red Hat's products.  In fact, the majority of the world's financial transactions are executed with Red Hat Enterprise Linux in the critical path.

This encompasses some of our industry's most mission-critical, performance-sensitive workloads.  We appreciate that the operating system needs to be secure, stable and performant...Full Stop.

That's what Red Hat's Engineering teams endeavor to deliver in each release, with Red Hat Enterprise Linux 7 being no exception.  In addition to Red Hat leading or assisting in development of features specifically demanded by the financial services industry, several years worth of research and development went into power management (i.e. c-states) research, kernel .config options, default sysctl tuning, kernel task scheduler and memory management, and network stack performance analysis.

Having personally worked very closely with stock exchanges for many years, I know that this level of tuning cannot be fully automated.  But we can always do better than defaults. New in RHEL7 is a tuned profile [editor's note:  "tuned" is pronounced tune-d] called 'network-latency'.  This profile lowers latency and improve determinism significantly as demonstrated by Tom Tracy in this video.

In recent weeks, Red Hat's Performance Engineering team has published an updated version of the popular Low Latency Tuning Guide for Red Hat Enterprise Linux 7.  This paper provides a tactical tuning overview on Red Hat Enterprise Linux 7 for latency-sensitive workloads on x86-based servers. It emphasizes impactful new features included in RHEL7, in areas such as CPU/power management, NUMA, tuned profiles, scheduling, network tunables and kernel timers. Additionally, it covers "de-jittering" CPU cores and tracing techniques to identify sources of jitter.

In contrast to the previous version of the paper written in 2012 for Red Hat Enterprise Linux 6, the updated version takes a simpler checklist approach covering prerequisite configuration steps to establish an environment optimized for low latency workloads.

• BIOS tuning
• NUMA topology
• PCI Locality
• Using the network-latency tuned profile
• Low-latency kernel command line

Subsequent sections provide context and supporting details on each step.

We hope this performance brief provides a helpful reference, and eases the tuning required to achieve the lowest possible latency and improved determinism on Red Hat Enterprise Linux 7.