When you write a program, you have to test it. Run, program, run! Did it do what you expected? Yay! Maybe you'll even set up a testsuite to run it many times, just to be sure. You might even create some sample files for it to work with.
What do you do when your "program" is your whole system? You can't just change your /etc config files just to test your new program, but that's exactly what you have to do when your "program" is the core C library, glibc. You can't even update the library easily, as the commands you use to update it (cp, mv) themselves rely on the library!
One option for this is to have a separate machine (real or virtual) set aside for testing. Still, installing a newly built glibc and automating tests on a remote machine is not a trivial task. Currently, most glibc testing is done in the "build directory" - an uninstalled glibc is tested, with options and hacks to force it to use its own support files and configuration. This, however, has some drawbacks. Consider this code flow:
When testing a non-installed glibc, you can test the "User API" portion of glibc's code, but you can't test the "System API" portion because it's bypassed - you end up only testing the "Test API" code, which is a waste of time as it will never get used by a real application. Ideally, we'd use the "System API" code for testing as well, but how?
Enter the Microcontainer
Linux Containers allow you to have a "system" that's just a subdirectory in your filesystem. Normally this subdirectory would be its own block device, with a full OS on it, managed by some high level software like Kubernetes or Docker. In our case, though, we only need "just enough" of a container to test glibc. Conveniently, glibc is at the core of the operating system, and has almost no dependencies. So, we can "install" glibc into an empty directory and consider that directory to be a container. We also only need the smallest amount of "containerizing" code as possible, making this the smallest implementation of a container. Hence, a microcontainer.
In glibc 2.28 I extended the testsuite's infrastructure to support such a microcontainer via a support program called test-container. The build system pre-installs a copy of glibc into an empty subdirectory, and test-container runs each test within that container. There are a few dummy programs we include that aren't part of glibc, like /bin/sh and /bin/echo, but since the container has its own filesystem the test now looks like this:
As you can see, the code path is simpler, and is the same path that real applications will use. The test-container program is able to isolate the test by giving it its own PID, UID, and filesystem namespaces. Each test that needs to run in a container may provide a skeleton set of files to install (like /etc/hosts) and/or commands to run (i.e. to chmod /etc/hosts) and the container is cleaned up between tests. Thus, the test may run as root, install "system" files, corrupt the environment, or whatever else it needs to do to test glibc.
The Benefits
Besides the obvious "more tests", this new feature also allows tests that would normally be run outside of glibc's testsuite to be migrated into it. A test that might have been run, say, by a QA department just prior to shipping a product with glibc in it, can now be run months if not years earlier by the glibc developers themselves. Tests also become less expensive to run if they previously required manual intervention, or the provisioning of an entire virtual machine. All this means that problems get found quickly, typically by the upstream developer working on that code, when the new code is fresh in their mind. This leads to a better upstream, which benefits all glibc users.
Last updated: September 3, 2019