Why Infrastructure Parity Matters for Developers Too

Background

I’ve been working with the CTO of an online video game company to develop a container architecture for his business. The goal is to simplify the deployment of new applications as well as make it easier to go back and change the code on older applications. The desired state is environmental parity across the infrastructure -- this will simplify the assignment of work on different applications to different developers. From developer laptops to production servers, the code will just work!

While video game production has unique technical and business requirements, infrastructure parity from developer laptops to production servers is a common desire that touches every industry that relies on application delivery.

An Interesting Problem

While discussing possible architectures something interesting came up - When doing video game programming, it is common to embed a scripting language into your main programming language. Lua happens to be a popular language for scripting video games. This company compiles their own Ruby interpreter because they need to embed Lua to script game play logic.

Following the LuaJIT installation instructions, they started on a development server that was set up like production. Ruby compiled without a problem and they began development. As they moved further into the development lifecycle, it became clear that it would be easier and more convenient to move most of the development work to the developer laptops.

So, they began the process of installing all of the developer tools on Mac OSX. Once complete, they began the process of compiling Ruby with LuaJIT. Everything compiled fine, but every time they ran the binary it would exit with a segmentation fault (segfault). The development team worked for four solid hours debugging. Eventually, they realized that they had missed a small piece of documentation.

From the LuaJIT installation guide

If you're building a 64-bit application on OSX which links directly or indirectly against LuaJIT, you need to link your main executable with these flags:

-pagezero_size 10000 -image_base 100000000

Also, it's recommended to rebase all (self-compiled) shared libraries, which are loaded at runtime on OSX/x64 (e.g. C extension modules for Lua). See: man rebase

Lesson Learned
Developers need to be thinking about the operating system user space. In fact, to avoid problems with moving to production, or setting up developer environments for new developers, I recommend two important things to developers. First, they run a virtual machine configured identical to production, even if it's just a container host - this can still prevent performance and security regressions. Second, and more importantly from a frustration perspective, develop with the exact same container images as are used in production. This consistency will speed things up and avoid 4-hour Yak shaving detours.

You might think you don't have this problem because you use Node.js, Java, Python, or vanilla Ruby right? Wrong - interpreted languages do not provide 100% isolation from the operating system. Though most non-video game developers are not compiling their own Ruby interpreter, almost every language has C based modules that do need to be compiled? Compiled modules are popular in Perl, Ruby, PHP, Python, and even Java has JNI. Even doing something as simple as "shelling out" with the system() function to execute a local shell script, can cause headaches when the user space is updated for security, performance, or support reasons.

To avoid these problems, you should run a constant container image (user space) and host (kernel) from development all the way to production. It just simplifies things over time and removes the chances of compatibility problems. At the end of the day, the CTO I worked with told me, “By using Containers, I can avoid this [compile problem] and create a documented and reproducible Linux environment. Containers let me develop an image on OS X and then deploy to Linux servers in production.” Infrastructure parity from developer laptops to production servers allows development teams to leverage the power of container images. These container images make it easy to avoid bugs created by user space differences between platforms. Let’s shave less Yaks and write more code!

Whether you are new to Containers or have experience, downloading this cheat sheet can assist you when encountering tasks you haven’t done lately.