by Jakub Hrozek and Andreas Schneider
Software testing is already a hard business. It gets even harder if you need to test software that is networked, requires custom users on the system or resolve DNS queries.
Consider software such as a file server -- it needs to listen for incoming connections on a certain port, often a privileged one in case of well-known protocols. The file server also requires the ability to switch to different user accounts and act on their behalf to create files owned by these users. Finally, a client of this hypothetical file server might want predefined SRV records to be present in DNS for autodiscovery to work properly. All these cases should be tested on every build.
And it gets even harder if your unit tests can't run as the root user to set up the environment. The use case of testing the full stack, including network, users or DNS with only regular user privileges is exactly what the cwrap.org project is aiming to solve.
It might seem that requiring root access for tests is not a blocker these days, where VMs are cheap and containers even cheaper. The obvious way to set up environment for full stack testing would be to create several VMs or containers where you'd have root access and that would provide you with the services you need, such as name resolution or custom user accounts.
This might work very well in an environment that you control, such as a development lab, but not in all environments that the tests might be required to run. For example, build systems often have no network interfaces at all and the builds run in a chroot, under a non-privileged uid. Also for developers, it is a huge time saver to be able to run tests from your normal user account while you develop and try a patch instead of submitting changes and waiting for results from a sophisticated Continuous Integration (CI) system. If you are an outside contributor to a project, you normally don't have access to the projects infrastructure, even if it exists. However it should be possible for a contributor to run the same test without setting up a test lab.
In comparison, the cwrap project allows you to set up the test environment locally on your machine, with just your account without any additional configuration of your system. In addition, this approach works on other UNIX operating systems like BSD or Solaris as well. The trick that the cwrap libraries use to allow the operations that would otherwise be privileged is called preloading.
What is preloading?
Preloading, often known by the name of the environment variable called LD_PRELOAD, is a feature of the dynamic linker. It lets you prepend a custom dynamic shared library before any other linked shared libraries and as a result, provide a custom implementation of any function call. For example, you might want to implement a library that has its own version of the the send() and recv() calls that print the data that pass through along with a precise timestamp before actually sending or receiving them.
That's exactly how the wrappers work -- they provide a preloadable version of publicly used system calls, typically from libc or similar low-level library that can be used in your test to simulate interaction with the system. In most cases, the preloadable versions of calls provided by the cwrap libraries return some data that can be set as part of setting up the test, or sometimes the data is routed through a different, local interface instead of a network. Since the libraries have no config file and no setup binary, they are mostly configured via environment variables.
All the libraries are part of the upstream cwrap project.
The cwrap project
cwrap.org is an umbrella project that contains several preloadable libraries for different purposes. While the fact that the libraries are developed in separate git trees and eventually released on their own is a relatively new project. The code itself has been present in the Samba project in a different form for several years already. With Samba being one of the most complex FOSS projects out there, implementing several networked protocols, testing was naturally a challenge, so Samba tests benefited from the wrappers a bit. After being developed internally in Samba for some time, the wrapper library code was recently cleaned up and released separately as a preloadable version by Andreas Schneider. Currently the wrappers all live at the cwrap.org site, where you can find information about the individual libraries and in addition check out the code. The following section will illustrate each of the wrappers in more detail along with an example.
The wrappers in detail
socket_wrapper
The socket_wrapper library simplifies testing of networked applications by routing all network communication over local unix sockets. While some deamons might be testable locally using some form of a local socket (like ldapi:// in case of LDAP), the codepath is usually different from what runs in production -- and then it's not the right test. Also, while many networked servers are able to bind to a local interface, chances are that a build system doesn't even have the loopback. On the other hand, using socket_wrapper is totally transparent to the application.
When an application that uses socket_wrapper is under test, the test defines a temporary directory. Every time the application opens a network socket (UDP or TCP), socket_wrapper intercepts the call via a preloadable function and routes the network communication into a UNIX socket in that temporary directory instead.
In addition, the socket_wrapper allows to dump all the traffic into a pcap file, which can be very useful for debugging. Using socket_wrapper simplifies networked tests in comparison with tests that would use multiple VMs or air-gapped machines by removing the step that sets up the networking. Instead, only the socket_wrapper setup is required instead, which is much simpler and as all wrappers also works locally or in a build root.
This example shows how the netcat utility can be tricked into communicating over local sockets only with socket_wrapper:
# Open a console and create a directory for the unix sockets.
$ mktemp -d
/tmp/tmp.bQRELqDrhM
# Then start nc to listen for network traffic using the temporary directory.
$ LD_PRELOAD=libsocket_wrapper.so SOCKET_WRAPPER_DIR=/tmp/tmp.bQRELqDrhM SOCKET_WRAPPER_DEFAULT_IFACE=10 nc -v -l 127.0.0.10 7
# Now open another console and start 'nc' as a client to connect to the server:
$ LD_PRELOAD=libsocket_wrapper.so SOCKET_WRAPPER_DIR=/tmp/tmp.bQRELqDrhM SOCKET_WRAPPER_DEFAULT_IFACE=20 nc -v 127.0.0.10 7
# (The client will use the address 127.0.0.100 when connecting to the server)
# Now you can type 'Hello!' which will be sent to the server and should appear in the console output of the server.uid_wrapper
The uid_wrapper provides preloadable versions of calls that tell the system who you are. Using uid_wrapper you can trick the system into thinking you are root or into switching credentials to different users. You can easily cause getuid to return root. Consider this shell example:
$ id
uid=1000(asn) gid=100(users) groups=100(users),478(docker)
$ LD_PRELOAD=libuid_wrapper.so UID_WRAPPER=1 UID_WRAPPER_ROOT=1 id
uid=0(root) gid=0(root) groups=0(root)Please note that using the wrapper would not magically make you root. The functions the appplication 'id' calls just returns the data we feed them via the preloaded library. Nonetheless, uid_wrapper is really useful to unit test code that can normally be ran only as the super user - some SSSD unit tests use uid_wrapper for precisely this reason!
nss_wrapper
The nss_wrapper allows the unit test to provide a custom passwd, group or hosts file to the application and by extension provide custom users and group entries to the application. Its functionality is required for unit tests that deal with users, groups and hosts, often in conjuction with uid_wrapper. For example, the SSSD unit tests leverages nss_wrapper to test the code that switches from root to the special daemon user after startup. A mechanism such as nss_wrapper is the only way to make sure the custom users are present in totally isolated environments.
Example:
$ echo "bob:x:1000:1000:bob gecos:/home/test/bob:/bin/false" > passwd
$ echo "root:x:65534:65532:root gecos:/home/test/root:/bin/false" >> passwd
$ echo "users:x:1000:" > group
$ echo "root:x:65532:" >> group
$ LD_PRELOAD=libnss_wrapper.so NSS_WRAPPER_PASSWD=passwd
NSS_WRAPPER_GROUP=group getent passwd bob
bob:x:1000:1000:bob gecos:/home/test/bob:/bin/falseThe second use-case of nss_wrapper is testing of NSS modules themselves. The unit test can set the prefix of the NSS module functions (which is typically inferred from the module name, i.e. nss_sss has the sss prefix) and load and call the module functions directly. The Red Hat QE team has unit tests that call nss_ldap functions this way.
resolv_wrapper
The newest addition to the cwrap.org family so far is resolv_wrapper. This library makes it possible to test applications that use the libresolv API -- a typical example would be a program that uses SRV DNS records to locate a server to connect to. This wrapper is diffent from nss_wrapper's host support that wraps the NSS calls such as getaddrinfo(). In comparison, the resolv_wrapper support the libresolv low-level API.
resolv_wrapper allows you to either set up a custom DNS server and point the test to this server with an environment variable or even set up a fake DNS database file that the application will use to construct test DNS replies. The first option is useful for testing DNS servers or integrated servers that include DNS, such as FreeIPA or Samba. The second option is more useful for testing clients where you need to test auto-discovery of services from a client.
A complete example - using uid_wrapper and nss_wrapper to test deamon code
Since this is a developer article after all, let's show some C code that illustrates the usage of two wrappers in synchronization. The following C code checks if its' running as root, if it is, switches into user bob, otherwise errors out. Our goal is to unit test the code:
#include
#include
#include
int main(void)
{
struct passwd *bob;
uid_t u;
int ret;
u = geteuid();
if (u != 0) {
fprintf(stderr, "Must be root to switch IDs!n");
return 1;
}
bob = getpwnam("bob");
if (bob == NULL) {
fprintf(stderr, "User 'bob' is not present on the systemn");
return 2;
}
ret = setresgid(bob->pw_gid, bob->pw_gid, -1);
if (ret != 0) {
fprintf(stderr, "Cannot set GIDn");
return 3;
}
ret = setresuid(bob->pw_uid, bob->pw_uid, -1);
if (ret != 0) {
fprintf(stderr, "Cannot set UIDn");
return 4;
}
if (geteuid() != bob->pw_uid || getegid() != bob->pw_gid) {
fprintf(stderr, "Expected to be bob now!n");
return 5;
}
printf("Became %d:%dn", geteuid(), getegid());
return 0;
}OK, let's first compile and run the program:
$ gcc wrapexample.c
$ ./a.out
Must be root to switch IDs!Not surprisingly, this didn't work, since we require root in our program. So the first step is to enable UID_WRAPPER and trick the code into thinking we're root:
$ LD_PRELOAD=libuid_wrapper.so UID_WRAPPER=1 UID_WRAPPER_ROOT=1 ./a.out
User 'bob' is not present on the systemThis got a bit further, but indeed, the special user bob is not present on this system at all, nor should it be. Therefore, we must also enable nss_wrapper:
$ LD_PRELOAD="libuid_wrapper.so:libnss_wrapper.so" UID_WRAPPER=1 UID_WRAPPER_ROOT=1 NSS_WRAPPER_PASSWD=passwd NSS_WRAPPER_GROUP=group ./a.out
Became 1001:1001Our test now passes! Notice how we used two libraries in the LD_PRELOAD environment variable value. This example resembles how many real-world programs drop privileges, so it's actually quite close to how cwrap can be used in your unit tests!
Real-world cwrap usage and future work
At the time of writing this article, several projects use the cwrap.org libraries. One is naturally Samba, that uses even the libraries introduced after the wrappers have been split from the Samba tree, like socket_wrapper. SSSD uses the wrappers to test its privilege separation code and has an integration test using the wrappers that starts an unprivileged LDAP server and fetches user accounts from the LDAP server using SSSD that is tricked by uid_wrapper into believing it's root. The socket_wrapper is also used by MIT Kerberos. Using the wrappers together with the cmocka unit testing library can bring very flexible test environment into your projects.
All the wrappers are highly portable. Linux is an obvious first-class platform, but the wrappers also work on several other operating systems such as FreeBSD, Solaris and MacOS X where applicable (for example MacOS doesn't have nsswitch.conf, so it can't support nss_wrapper). The cwrap developers also take pride in testing their software. All wrappers have very high code coverage, so you can be sure the code works.
As a next project, the cwrap developers would like to make it possible to test PAM applications better, via another wrapper called pam_wrapper. Code contributions, ideas or bug reports from users and developers are very much welcome as well!
Links
The cwrap.org homepage, includes links to all the sub-projects
https://cwrap.org
SSSD unit test leveraging uid_wrapper and nss_wrapper:
https://git.fedorahosted.org/cgit/sssd.git/tree/src/tests/cwrap/test_become_user.c
The cmocka unit testing framework
Last updated: August 31, 2016