Docker project: Can you have overlay2 speed and density with devicemapper? Yep.

It’s been a while since our last deep-dive into the Docker project graph driver performance.  Over two years, in fact!  In that time, Red Hat engineers have made major strides in improving container storage:

All of that, in the name of providing enterprise-class stability, security and supportability to our valued customers.

As discussed in our previous blog, there are a particular set of behaviors and attributes to take into account when choosing a graph driver.  Included in those are page cache sharing, POSIX compliance and SELinux support.

Reviewing the technical differences between a union filesystem and devicemapper graph driver as it relates to performance, standards compliance and density, a union filesystem such as overlay2 is fast because

  • It traverses less kernel and devicemapper code on container creation (devicemapper-backed containers get a unique kernel device allocated at startup).
  • Containers sharing the same base image startup faster because of warm page cache
  • For speed/density benefits, you trade POSIX compliance and SELinux (well, not for long!)

There was no single graph driver that could give you all these attributes at the same time — until now.

How we can make devicemapper as fast as overlay2

With the industry move towards microservices, 12-factor guidelines and dense multi-tenant platforms, many folks both inside Red Hat as well as in the community have been discussing read-only containers.  In fact, there’s been a –read-only option to both the Docker project, and kubernetes for a long time.  What this does is create a mount point as usual for the container, but mount it read-only as opposed to read-write.  Read-only containers are an important security improvement as well as they reduce the container’s attack surface.  More details on this can be found in a blog post from Dan Walsh last year.

When a container is launched in this mode, it can no longer write to locations it may expect to (i.e. /var/log) and may throw errors because of this.  As discussed in the Processes section of 12factor.net, re-architected applications should store stateful information (such as logs or web assets) in a stateful backing service.  Attaching a persistent volume that is read-write fulfills this design aspect:  the container can be restarted anywhere in the cluster, and its persistent volume can follow it.

In other words, for applications that are not completely stateless an ideal deployment would be to couple read-only containers with read-write persistent volumes.  This gets us to a place in the container world that the HPC (high performance/scientific computing) world has been at for decades:  thousands of diskless, read-only NFS-root booted nodes that mount their necessary applications and storage over the network at boot time.  No matter if a node dies…boot another.  No matter if a container dies…start another.

Continue reading “Docker project: Can you have overlay2 speed and density with devicemapper? Yep.”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Take advantage of your Red Hat Developers membership and download RHEL today at no cost.


For more information about Red Hat OpenShift and other related topics, visit: OpenShift, OpenShift Online.

Tuned: the tuning profile delivery mechanism for RHEL

What is “Tune-D” ?

Tuned is a tuning profile delivery mechanism included in Red Hat Enterprise Linux.  As demonstrated by D. John Shakshober (aka Shak) at Red Hat Summit, tuned improves performance for most workloads by quite a bit.  What’s a tuning profile, you ask?  Using the throughput-performance profile (enabled by default in RHEL7) as an example:

tuned-throughput-performance

 

These settings tune RHEL for the datacenter, whether public cloud, or private.  You can easily create your own profiles, too!

Red Hat delivers tuned profiles for most of our product portfolio:

Continue reading “Tuned: the tuning profile delivery mechanism for RHEL”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Can you run Intel's Data-plane Development Kit (DPDK) in a Docker container? Yep.

nicubunu_PackageAs part of our participation in hundreds of open source communities, Red Hat engineers are often involved in research and development efforts that may or may not become a part of Red Hat’s supported offerings.

Intel’s Data-plane Development Kit (DPDK) is a set of libraries and drivers for Linux and BSD built for fast packet processing, for the burgeoning “Network Function Virtualization“, or NFV discipline.  Typical verticals interested in turning Linux boxes into packet-processing machines are telecom, financial services, military, energy research, datacenter operators, internet service providers and many more.

Continue reading “Can you run Intel's Data-plane Development Kit (DPDK) in a Docker container? Yep.”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Accelerating Red Hat Enterprise Linux 7-based Linux Containers with Solarflare OpenOnload

RH_Icon_Container_with_App_FlatLinux Containers combine well-established Linux kernel technologies such as namespaces, SELinux, cgroups and iptables with incredible ease of use and exceptional performance.

For customers looking for the lowest possible network latencies and reduced CPU overhead coupled with the deployment advantages of Linux containers, Red Hat’s new Accelerating Red Hat Enterprise Linux 7-based Linux Containers with Solarflare OpenOnload whitepaper provides installation, configuration and tuning guidance for Docker containers running on Red Hat Enterprise Linux with Solarflare OpenOnload network acceleration.

Continue reading “Accelerating Red Hat Enterprise Linux 7-based Linux Containers with Solarflare OpenOnload”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Shaping the Performance of a Linux Distro: Inside Red Hat Enterprise Linux 7

Red_Hat_RGB_150px

Backstory

Red Hat’s Performance Engineering team is responsible for the performance of many of Red Hat’s products.  We cover existing

products such as RHEL, OpenStack Platform, OpenShift and RHEV, as well as newer products like Ceph and CloudForms.

Although these days we contribute extensively to Red Hat’s cloud offerings, Red Hat Enterprise Linux remains a core responsibility as the building block for our large ecosystem of customers and partners, plus much of Red Hat’s growing product portfolio.

Smoketest Surprise

Some our earliest work on what would become RHEL7 was done on 3.5-ish kernels (somewhere in the Fedora 17 timeframe).  As more and more results started to roll in, we became a bit concerned about the performance of some aspects of the upstream kernel.

While some results were on-par with RHEL6, many were not.  In fact, some OLTP workloads regressed by 20+%, and some latency benchmarks were off double-digits, too!  It was clear that we had our work cut out for us.  And that leads us to…

How we spent 2012 and 2013

The years 2012 and 2013 were spent in the lab making nerdy signs…

Notre Dame Sign

…scoping and characterizing the earliest builds of what would become RHEL7.

Fun stuff like crushing (and crashing!) upstream kernels, and working with partners and customers to ensure a successful release of the newest version of Red Hat Enterprise Linux, version 7.0.  As a company, easily our most important release because not only must it delight our existing RHEL customer base, but it must also become the foundation of our cloud portfolio.  We had to get it right.

As some of the initial evaluators of upstream or test code, our most significant (mostly hidden) contribution to the open source community is providing early adopter feedback to our development teams during the design phase, when code is most flexible.  We need to know things like:

  • How it’s designed.
  • Where are the long poles, and can they be mitigated?
  • What makes sense for defaults?
  • Do we need tracepoints — how do I observe the critical sections under load?
  • Finally, a personal goal…How do we avoid surprising the sysadmins, who are again further downstream, and the very lifeblood of a infrastructure provider like Red Hat.

One of the key discoveries we made as a team throughout a decade of supporting customers on RHEL, was an obsession the prioritization of customer experience.  We knew we wanted our customers to be absolutely thrilled with by the stability, reliability, security and most importantly 😉 the performance of Red Hat Enterprise Linux 7.

To the way-back machine…

We’d actually begun the journey towards workload-specific tuning in RHEL5 with the introduction of ktune.  ktune provides some nascent tuning profiles for a very small set of workloads.  We did not expand much on ktune in the RHEL5 product.

evolution

With the GA of RHEL6.0, we introduced the tuned package.  I like to describe tuned as a “tuning profile delivery mechanism”, and ends up being our group’s primary feedback loop into the Red Hat’s product line.  If you haven’t heard of tuned in RHEL6, trying it is as simple as:

# yum install tuned
# service tuned start
# tuned-adm list
# tuned-adm profile throughput-performance

We received lots of great feedback from our partners and customers about performance in RHEL6.  Coupled with additional R&D, this feedback allowed us to confidently expand the reach of tuned and it’s profiles beyond RHEL.  So in addition to the profiles shipped with RHEL, Red Hat began to ship profiles for a growing list of products such as RHEV, RHEL OpenStack Platform, OpenShift, RHEL Atomic and Red Hat Storage.

Here’s a chart depicting the tuned profile inheritance feature in RHEL7.  RHEL includes 3 “parent” profiles, a handful of child profiles, and even some grand-children.  Users are free to customize existing profiles, create their own, or use none at all.

tuned-inheritance

Based on extensive testing, these profiles typically boost performance in the double-digit percent range “for free”.  Just some of the benefits of Red Hat’s Enterprise-hardened distribution versus free/community software.

Time to make the donuts RHEL7

Through 2+ years and 4+ minor releases of RHEL6, our team gained valuable experience with the tuning profiles customers began adopting.  As we began working on RHEL7, the next logical step to improve out-of-the-box performance for our customers, was to meticulously validate upstream Linux kernel defaults through the lens of enterprise datacenter and cloud environments.

Although it’s impossible to identify one specific set of tunings that help all workloads, this effort led us to a set of changes to default kernel compile-time settings and sysctl tunings that would boost performance of most workloads well passed upstream defaults.  These changes were so impactful, that we collaborated with our kernel engineering teams to further validate, and propose tuned be enabled by default in RHEL7.  After many months of discussion and testing, this proposal was accepted and delivered in the RHEL7.0 GA.

Note:  as a further optimization, tuned will automatically customize itself based on what version of RHEL (Workstation/Server) is installed, and whether it’s running on virtualization or bare metal.

Summary

Prior to beginning efforts on RHEL7 in earnest, Red Hat’s Platform Business Unit continued to deliver minor updates to RHEL5 and RHEL6 simultaneously, along with extending support to 10 years.  Concurrent delivery of 3 major streams of RHEL is only possible because of the depth of talent (and frankly, depth of character) present in our engineering teams.

As a fairly large team of performance engineers, we remain focused on ensuring RHEL continues to be the trusted, rock-solid foundation upon which customers and partners should remain confident to run their business.


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Take advantage of your Red Hat Developers membership and download RHEL today at no cost.

Introducing the "rhel-tools" for RHEL Atomic Host

RH_Icon_Container_with_App_FlatThe rise of the purpose-built Linux distribution

Recently, several purpose-built distributions have been created specifically to run Linux containers.  There seem to be more popping up every day.  For our part, in April 2014 at the Red Hat Summit, Red Hat announced its intention to deliver a purpose-built, container-optimized version of Red Hat Enterprise Linux 7 called RHEL Atomic Host.  After over a year in the making, we are excited that launch day has finally come!

What’s important to know about Red Hat Enterprise Linux Atomic Host, you ask?  Well, plenty…but for the sake of this blog, I’ll stick to areas I know as a performance engineer:

  • RHEL Atomic leverages years of engineering effort that went into RHEL7.
  • It uses the same exact kernel as RHEL7.
  • Significantly reduced on-disk and in-memory footprint.
  • Utilizes OSTree technology for upgrades and rollbacks.
  • Optimized device-mapper container storage performance out of the box.
  • Optimized container scalability out of the box.
  • Includes purpose-built rhel-tools container for system administration tasks

Continue reading “Introducing the "rhel-tools" for RHEL Atomic Host”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Low Latency Performance Tuning for Red Hat Enterprise Linux 7

velocimetroCounting micro-nanoseconds?  We are, because we know our customers are.  Some of the world’s largest stock exchanges including the Chicago Mercantile Exchange (CME), New York Stock Exchange (NYSE), E*TRADE, Union Bank, countless hedge funds and high-frequency trading shops run on Red Hat’s products.  In fact, the majority of the world’s financial transactions are executed with Red Hat Enterprise Linux in the critical path.

Continue reading “Low Latency Performance Tuning for Red Hat Enterprise Linux 7”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Take advantage of your Red Hat Developers membership and download RHEL today at no cost.

Beyond Microbenchmarks: breakthrough container performance with Tesla efficiency

Back story

As virtualization was beginning it’s march to prominence, we saw a phased approach to adoption.  This is common with any sort of game changing technology….let’s take electric cars as an example.  Early adopters are willing to make certain trade-offs (short range) to gain new capabilities (saving money at the gas station).

teslaIn the meantime, engineers are off in the lab working hard to increase the possible consumer-base for electric cars by increasing range, decreasing charging cycle times, and improving performance.  Taken in aggregate, those changes are meant to address objections to the first-cut of the technology.

Virtualization is to Linux containers is to…

Continue reading “Beyond Microbenchmarks: breakthrough container performance with Tesla efficiency”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Take advantage of your Red Hat Developers membership and download RHEL today at no cost.

Comprehensive Overview of Storage Scalability in Docker

Backstory

First, a brief backstory on the storage situation for Docker since it was open-sourced in early 2013.  At that time, Docker relied on a filesystem called AUFS (advanced multi layered unification filesystem).  This Union filesystem provided the necessary features to support several of Docker’s main selling points:

homepage-docker-logo

Continue reading “Comprehensive Overview of Storage Scalability in Docker”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!

 

Performance Analysis of Docker on Red Hat Enterprise Linux 7

Containers introduce some intriguing usability, packaging and deployment patterns. These new patterns offer the potential to effect massive improvements to the enterprise application development and operations specialties. Containers also offer the promise of bare metal performance while offering some amount of isolation as well.

But can they deliver on that promise ?

Continue reading “Performance Analysis of Docker on Red Hat Enterprise Linux 7”


Join Red Hat Developers, a developer program for you to learn, share, and code faster – and get access to Red Hat software for your development.  The developer program and software are both free!