Red Hat Summit: Building production-ready containers

Bringing excitement to the last session on the last day of the show, Scott McCarty and Ben Breard wrapped up this year's Red Hat Summit with a discussion of best practices for production-ready containers.

In the container era, Scott pointed out there are four building blocks you need to think about:

• Container images
• Container hosts
• Container orchestration
• Registry servers

Each of these topics is a huge rabbit hole you can go down if you want to really learn all there is to know about them. As you'd expect from the session title, Scott and Ben focused on container images. Despite that, you still have to consider the other three. How will your images interact with others? How will you get data to them? How will they interact with each other? Will you embed passwords in your images? (Spoiler alert: No.) You need to take all of these things into consideration as you move into the world of containers.

A single container is about as useful as a single Lego block¹. You need to tie lots of them together in interesting ways to get the full power of containers behind you. Scott quoted Red Hat's Ryan Hallisey:

Using containers is as much of a business advantage as a technical one. When building and using containers, layering is crucial. You need to look at your application and think about each of the pieces and how they work together–similar to the way you can break a program up into classes and functions.

If you're building a Lego model, you take an instruction sheet and a set of building blocks and create something from them. In the world of containers, you take instructions (YAML) and building blocks (images) and create an application from them.

The goal of the Open Container Initiative (OCI) is to define standards for containers and runtimes. (Members include Red Hat, CoreOS, and pretty much every other player in the industry.) If you're an architect, OCI protects your investment because you can create images once and know you can use them for the foreseeable future² and know that the tooling and the distribution and logistics mechanisms and the registry server will still exist and will still work.

As the old joke goes, the great thing about standards is that you have so many to choose from³. Scott mentioned five relevant standards that are being driven by vendors, communities, and standards bodies:

• The OCI Image Specification
• The OCI Distribution Specification
• The OCI Runtime Specification
• The Container Runtime Interface (from Kubernetes)
• The Container Network Interface (from the Cloud Native Computing Foundation)

When you're running a container, there's an image, there's a registry, and there's a host. With the Distribution spec, you're protected if you put your images in an Amazon registry, an Azure registry, a Red Hat-provided registry, etc. You can move that image around and run it in another environment.

Scott mentioned a number of tools that are emerging as the standards become more entrenched. The community created crictl for the CRI standard. Podman (now available as a tech preview) offers an experience similar to the Docker command line. runc is a command-line tool to run containers according to the OCI runtime spec. Project Atomic created the Buildah tool to build OCI-compliant images. What's great about Buildah is that it will work with your Dockerfiles. Buildah lets you add packages into the image when you build it, so that your final container doesn't have to have a package manager. And of course you can use these tools in a toolchain.

Scott made the point that the Docker commands related to images actually work with repositories, not images. For example, this command:

docker pull registry.access.redhat.com/rhel7/rhel:latest

goes to the registry at registry.access.redhat.com, then looks in the rhel7 namespace, then looks for a repository named rhel, then takes the version of that image with the tag latest⁴. The hierarchy here is registry server/namespace/repo:tag. The registry server is resolved by DNS, but the namespace can mean different things depending on the registry server. At registry.access.redhat.com, the namespace is the product name. At Dockerhub, the namespace is the name of the user who committed the image. In creating images in your own registry, it's important to think through what the different components of the name mean. How will your organization use namespaces? How will you name your repos? (A bit of advice from Scott: always use the full URL of the image in your Dockerfile to make sure you're getting exactly what you expect.)

Note: If you'd like to get a better handle on registries vs. repos and develop a deeper understanding of containers, check out Scott's article: A Practical Introduction to Container Terminology. That article, published in February of 2018 on developers.redhat.com/blog, is a complete update of Scott's popular article from 2016.  The 2018 version includes a lot of what is happening in the world of containers beyond docker.

Next Ben took over to discuss his tenets for building images. He stressed that you should use source control for everything so that all of your artifacts are buildable from code. His five basic principles for building production-ready containers are:

• Standardize
• Minimize storage
• Delegate
• Process
• Iterate

We'll discuss these over the next few paragraphs.

Standardize: Your goal should be to have a standard set of images with a common lineage. Your base images will be things like application frameworks, app servers, databases, and other middleware. The obvious benefits here are that your images are easier to scale, reuse of common layers is maximized, and the differences between environments in your various containers are minimized. The size of registries can be a problem, especially with thousands of developers constantly cranking out images as toolchains and build pipelines do their magic. Standardizing on as few images as possible has huge benefits in your registry, at runtime, and whenever you need to update a base image. (Red Hat encourages you to use our base images, especially the LTS images.)

Minimize storage: The goal is to limit the content in a given image, particularly a base image, so that it only has what you're using. Red Hat provides an image named rhel7-atomic (not to be confused with Project Atomic). This image has glibc and just enough rpm to add packages. There is no python, no systemd, or similar things that you probably don't need. Remember, with an image, you're building a sandbox. If your sandbox is the size of a stadium parking lot, it's not a sandbox anymore.

Delegate: Ownership of an image should lie with the people who have the most expertise for that image, whatever it contains. Don't be a hero; don't be responsible for every image in your organization. Leverage your team's skills.

Focus on process and automation: This is the most important rule. The barrier to getting started with containers is really low. It's not that much trouble to create a Dockerfile and run docker build. That means it's simple to do something once and never think about it again. But containers are not "fire and forget." You need process around them for everything from testing to deployment to security. Ben mentioned tools like OpenSCAP and Clair that can scan your images for vulnerabilities.⁵

Iterate is the final goal. Don't repeat the mistakes of the past. Making changes is no longer a big deal. If you have testing and security scanning as part of your build chain, then you go from "known good" to "known good" every single time.

The image a developer has on her laptop should be the same as the image in the dev/test environment, which should be the same as the image in production in the cloud. (BTW, the term "developer" here really means "anybody who builds an image." That could include sys admins or architects or others.) As you distribute an image, the YAML file that defines persistent volumes, secrets, scaling policies, and other metadata should go along with it as well.

Finally, Ben made the point that you need to be building images in a system that has a pipeline. You can build images on your local machine for smoke testing, but any changes that are significant should be run through the pipeline. It's crucial to remember that you can't just fire up Docker on a laptop and test your image. To do test anything significant, you'll need an environment that can start a Kubernetes cluster, pull down all the images that run together in production, and then start the pipeline. Nobody wants to run a whole orchestrated system on their laptops, but that's the only way you can reliably test a set of services that work together.

The hallmark of a great piece of code is not just that it works, but that its architecture is elegant, intuitive, and flexible. Most experienced developers know how to do that. As we go forward, the ability to create a set of elegantly composed containers will be an essential skill. The right amount of functionality in each container, hierarchies of containers that simplify changes and upgrades, and the appropriate use of pipelines are all part of a production-ready, containerized application.

All in all, a great session with lots of best practices and good advice from two highly experienced speakers. If you'd like to hear all the details, the video of Scott and Ben's presentation is one of the 100+ Red Hat Summit 2018 breakout sessions you can view online for free

https://youtu.be/nizud-1IK9c]

¹ Containers on a container ship fit together like Lego blocks, if you think about it. Makes the metaphor even stronger.

² Which is maybe a couple of years. Seriously, how many things do you do today that you didn't do in 2015?

³ I didn't say the joke was funny, I just said it was old. Sorry about that.

⁴ Keep in mind that the tag "latest" is just a convention. It may or may not point to the most recent version of the image.

⁵ Ben recommended reading The Phoenix Project as a cautionary tale.