OpenSSL is an essential library for securing web traffic. This article offers simple procedures for initializing and terminating applications using OpenSSL. Modern applications that are more complex than "Hello, world!" usually require several external libraries like OpenSSL, which in turn often need to be properly initialized on startup and deinitialized on shutdown.
OpenSSL libraries are set up internally during program initialization. At this time, they load the configuration file, allocate resources, and handle FIPS mode, among many other tasks.
The OpenSSL API function for initialization is OPENSSL_init_crypto. This function accepts a variety of options with reasonable defaults. Initialization should be performed before any other OpenSSL function is used, though some OpenSSL functions invoke OPENSSL_init_crypto themselves.
Cleaning up in an atexit function
When you shut down your application, you need to free the resources you acquired. Because developers don't always code accurately, OpenSSL relies on a function provided by the C runtime library named atexit. This function specifies a cleanup function that will be executed upon normal termination of the application.
atexit is useful for many purposes, so it is widely used. Even though most resources are freed on any program termination, it's always better to free them explicitly–for example, to prevent false positives from being turned up by Valgrind or similar tools searching for memory leaks.
The problem with atexit is that real-life applications use many different libraries, each of which might install its own cleanup function. Although the atexit man page declares that the registered functions are called in the reverse order of their registration, the real order of execution is not predictable in practice.
One obvious scenario where the order is undefined is a multithreaded setup. Various plugins and third-party libraries might have their own atexit handlers that can be loaded at runtime, which can be another possible source of problems.
Executing cleanup functions in the wrong order can cause a crash on application shutdown. For example, an application might establish an SSL connection and create the corresponding objects. On shutdown, if the SSL object is deleted before the OpenSSL cleanup, everything is OK, but otherwise a crash will occur. This example is a real bug found recently by Nmap users. Originally reported for Solaris, it was later reproduced on a Fedora machine. If you find problems in a specific application or library, it's worth filing a bug in the hope that the problem will be fixed.
Problems with atexit handlers are reported to the upstream OpenSSL developers, Red Hat, and Fedora developers on a regular basis. There are several ways to solve the problems.
The authors of Nmap resolved the reported issue by using one more function from the OpenSSL API: OPENSSL_atexit. This function registers an application-specific handler, which is a suitable location for ensuring the order of resource deinitialization. The approach used in Nmap could cause a memory leak in some circumstances, but because it happens during shutdown, that isn't too much of a big deal. However, we don't recommend this option because memory leaks on shutdown make it more difficult to search for important runtime memory leaks. Using OPENSSL_atexit also makes code more complex.
Recommended initialization and shutdown procedure
The best option is a properly serialized deinitialization, during which the atexit hooks are executed in the correct order, all the application threads terminate except the main one, etc. This sequence requires you to be thorough in your coding, but it's worth the effort.
If your application has to shut down manually, OpenSSL authors suggest a procedure with two parts.
First, the application should use the OPENSSL_INIT_NO_ATEXIT option in the OPENSSL_init_crypto call at OpenSSL initialization, to avoid implicit installation of the OpenSSL cleanup handler. After that, when the application terminates and no further OpenSSL functions will be called, invoke OPENSSL_cleanup explicitly to free resources on shutdown.
In conclusion, a manual shutdown option is useful for some libraries, but a predictable order of initialization and deinitialization is preferable.