Accelerate glibc test development with the glibc-support repository

Over the years, the GNU C library (glibc) has collected library of helper functions in its upstream source tree, for writing in-tree glibc tests that run as part of its test suite. Like the test suite itself, this part of glibc is not installed, and is not available for general application development. This means that test development usually has to happen within the glibc source tree, which poses some challenges. However, there is an alternative, the glibc-support repository, which allows out-of-tree development of glibc test cases. Often, this simplifies development of test cases (particularly debugging), allows quicker iteration, and more efficient test case creation. Test cases produced this way can be integrated easily into the upstream test suite.

What is available in the glibc support facility?

Use of test support facility (so named after the support/ subdirectory in which it resides in the upstream source tree) is usually a condition for upstream acceptance of a new test. At the minimum, this means that the test routine is called do_test, and #include <support/test-driver.c> is used at the end of the test source file to arrange for calling the do_test function.

However, besides this formality, there are a broad range of facilities that help with writing tests in general, focusing on the core logic instead of verbose error. Some specialized functionality exists as well. The following list provides a broad overview of what is available.

• Timing out tests, using a test driver or wrapper. This was the starting point, back when #include "../test-skeleton.c" was still written at the end of test source files.
• Robust overall test failure detection. A shared memory segment records failure status, which means that even after a fork, a test failure in the subprocess does not go unnoticed.
• Checking and reporting helpers in <support/check.h>. For example, there is a TEST_COMPARE_STRING macro for comparing two strings, and on mismatch, the source location is printed, along with the two strings.
• Error-checking wrappers for commonly used glibc functions. These terminate the process on failure, instead of returning an error. Sometimes, this simplifies the programming interface beyond the removal of error checking code from the test proper. For example, xpthread_create returns the thread handle directly, while the original pthread_create function writes it to a variable whose address has been passed to pthread_create.
• Special allocation functions. There is support_allocate_shared to allocate memory that is shared with subprocesses, support_next_to_fault_allocate producing an allocation that is guaranteed to be followed by an unmapped page, and support_blob_repeat_allocate to create huge repeated strings that require relatively little physical memory because they use alias mappings. (This is useful to test if 64-bit targets can handle input strings longer than INT_MAX, for example.)
• An in-process fakeroot variant based on Linux user namespace support called support_become_root. This allows running tests which require chroot (to test functionality which involves hard-coded absolute paths) as regular users during development.
• A more complete test-container helper. It can be used to run tests in a container environment with a properly installed glibc, again based on Linux namespace support. (This is a direct implementation, using only glibc and kernel functionality.)
• A framework for writing DNS servers, to test the DNS stub resolver. This began as a regression test for CVE-2015-7547, but has since been used to verify fixes for several other defects, and to test new stub resolver features as well.
• Hopefully soon, a FUSE-base facility for writing file system tests.

Why in-tree glibc test development can be challenging

Not every developer is comfortable with working with the glibc sources directly.

The glibc project is not small, and build times can be considerable. There is a shortcut to build and run just one test (make t=support/tst-support_quote_string test) and all the tests in a set of subdirectories (make -j`nproc` subdirs=support check). However, if you edit one of the library (not test) files, and forget to rebuild everything (using make -j`nproc) before building the test, it is possible that your build tree ends up in an inconsistent state, and you have to delete it, and build again from scratch.

Due to the complexity of the glibc build system, tight IDE integration is usually not available.

Debugging support can be limited. In general, it is possible to perform source-level debugging using GDB if glibc was configured with ./configure --prefix=/usr --enable-hardcoded-path-in-tests. Without it, an unfortunate interaction between GDB and glibc makes it hard to debug the test sources themselves. However, once --enable-hardcoded-path-in-tests is used, the testrun.sh helper script can no longer be used to run existing (out-of-tree) programs against the new glibc.

If working with glibc sources that are newer than the system-provided glibc, it can happen that the new test does not run against the system version of glibc. Of course, this is unavoidable if the test targets functionality which is not yet present in the system version of glibc. But when writing regression tests, or just trying to reproduce a particular bug in an isolated test case, it can be very useful to develop the test against the system glibc version.

Similarly, during test development, it may be desirable to link against a system library. Doing so within the glibc build framework may fail or may not result in the intended result.

Using the glibc-support repository

Start by cloning the glibc-support repository:

https://pagure.io/glibc/glibc-support

In the glibc-support directory, build the sources:

$ make
…
g++ -O2 -fPIC -Wall -g -Werror=implicit-function-declaration -o build/tst-example-c++  build/tests/tst-example-c++.o build/libsupport.a 
rm build/tests/tst-example-c++.o build/tests/tst-example.o
$

After this, C source files named tests/tst-*.c will automatically be built as tests and linked against the glibc support facility. Similarly, C++ tests will be built if their sources are in files matching tests/tst-*.cc.

A minimal test using the glibc test driver looks like this:

static int
do_test (void)
{
  return 0;
}
#include <support/test-driver.c>

You need to save this source file under a name such as tests/tst-minimal.c, so that the build system picks it up. After running make, the resulting test program will be build/tst-minimal. You can run it directly from the shell:

$ build/tst-minimal
$ echo $?
0

What happens if there is a test failure? The next example uses the FAIL macro from <support/check.h> to report a test failure. It does so from a subprocess, but the test framework is still able to report the error back to the original process, although the subprocess exits with status 0, indicating success (support_isolate_in_subprocess is just a wrapper around fork and waitpid, running the callback function in a separate process):

#include <stddef.h>
#include <stdio.h>
#include <support/check.h>
#include <support/namespace.h>
#include <unistd.h>
static void
callback (void *closure)
{
  FAIL ("intentional failure (PID %d)", (int) getpid ());
  _exit (0);
}
static int
do_test (void)
{
  printf ("info: original test PID: %d\n", getpid ());
  support_isolate_in_subprocess (callback, NULL);
  return 0;
}
#include <support/test-driver.c>

This file should be saved as tests/tst-failure.c. Running the test program after make correctly reports the failure:

$ build/tst-failure 
error: tst-failure.c:9: intentional failure
error: 1 test failures
$ echo $?
1

One thing to keep in mind is that the glibc-support repository still uses the glibc test wrapper. This means that by default, the process forks on start, to install the timeout handler. GDB will not observe execution of the do_test function because it happens in a subprocess:

$ gdb --quiet build/tst-minimal
Reading symbols from build/tst-minimal...
(gdb) b do_test
Breakpoint 1 at 0x4024a0: file tst-minimal.c, line 4.
(gdb) r
Starting program: …/build/tst-minimal 
[Thread debugging using libthread_db enabled]                                   
Using host libthread_db library "/lib64/libthread_db.so.1".
[Detaching after fork from child process 69782]
[Inferior 1 (process 69779) exited normally]
(gdb) 

The easiest way to address this is to run the test with an additional --direct argument, which skips the timeout handler and the initial fork:

$ gdb --quiet --args build/tst-minimal --direct
Reading symbols from build/tst-minimal...
(gdb) b do_test
Breakpoint 1 at 0x4024a0: file tst-minimal.c, line 4.
(gdb) r
Starting program: …/build/tst-minimal --direct
[Thread debugging using libthread_db enabled]                                   
Using host libthread_db library "/lib64/libthread_db.so.1".
Breakpoint 1, do_test () at tst-minimal.c:4
4     return 0;
(gdb) 

Except for the --direct argument, there is nothing special about the test program, so you can run it using strace, valgrind or any debugging tool of your choice, just like any other program.

Contributing the test upstream

After the test works (or fails) as expected outside the glibc source tree, it is time to integrate into the upstream sources. The steps are roughly as follows:

• Pick an appropriate subdirectory, usually where the source file for the tested functionality resides. For example, for mkstemp, it would be the misc/ subdirectory (not stdlib/, even though mkstemp is declared in the <stdlib.h> header). So in this example, the test could be called misc/tst-mkstemp.c. If the test is Linux-specific, use sysdeps/unix/sysv/linux/ as the directory.
• Add the test to the Makefile file in that directory. Usually, there is a tests += \ line, and the test needs to be added to the sorted list after it. If editing sysdeps/unix/sysv/linux/Makefile, add it under the relevant subdirectory check, such as `ifeq ($(subdir),misc).
• In an update-to-date build tree (maybe after running make -j`nproc`), invoke make t=misc/tst-mkstemp to build and run the new test. There should be a PASS: line in the output.
• Run the tests for the entire subdirectory: make -j`nproc` subdirs=misc check. The new test should not appear in the output (neither under FAIL, XFAIL, nor under UNSUPPORTED).
• Commit the new test locally with an explanatory commit message. Include Signed-of-by: only if you are submitting under DCO.
• Clone the local sources, build them, and run the entire test suite, perhaps using git clone . CLONE && cd CLONE && mkdir build && cd build && ../configure --prefix=/usr && make -j`nproc` && make -j`nproc` check
• Post the patch to libc-alpha@sourceware.org.

Limitations of the glibc-support repository

Not surprisingly, only things available in the system glibc version (or part of the support facility) can be tested this way. However, it is possible to add additional (non-test) source files in the support-extra/ subdirectory of the glibc-support repository. This way, new functionality can be tested as long as it itself can be built outside of the glibc source tree.

Not all support functionality is available in the separate repository. For example, the DNS test framework depends on an internal glibc function, __res_iclose, and that is why it is currently not available outside the glibc build tree. The test-container helper is not built either. Instead you can use Mock, pbuilder, or more recent container tooling such as Podman.

Most of the remaining limitations are caused by the lack of traditional glibc Makefile integration, and that test source files are picked up for building automatically.

Building shared objects is not supported directly. This can be problematic if the goal is to create a dynamic linker test.

It’s currently not possible to control compiler and linker flags used to build tests. The CFLAGS and LDFLAGS environment/make variables are recognized, though, so it is possible to set the flags when running make to build the tests.

Environment variables to be used during test execution need to be specified manually, when the test is run. The glibc-support framework currently does not provide a way to apply them automatically, or any facilities for running tests—only for building them.

Conclusion

The glibc-support repository provides a convenient way to develop a wide range of tests for glibc. As discussed, there are some limitations, but in return the out-of-tree test builds enable much quick iteration during test development, and debugging is much simplified.

Try it and write a glibc test today.