How to use RHEL application compatibility guidelines

In this three-part series, we explore the Red Hat Enterprise Linux published application compatibility guidelines (ACG), and how developers can use them to ensure their application remains compatible with future releases of RHEL. Building applications can be difficult, and building applications that continue to operate after an in-place distribution upgrade is even harder. How does Red Hat Enterprise Linux make it easier? It provides guidelines and guarantees that you can follow to improve application compatibility.

In this article, we will expand on the concept of application compatibility. In the second part, we will review the topic of compatibility with more examples. In the third article, we will discuss container userspace compatibility with the host kernel services.

What is application compatibility?

What we call application compatibility is traditionally referred to as backwards compatibility. It is the ability to run an unmodified application binary on the current or newer version of the distribution and have it operate correctly (i.e., compatible with the release of the distribution).

Maintaining compatibility

Application compatibility is maintained by ensuring that the dependencies of the application continue to be provided and that the application continues to function as intended.

When we talk about C or C++ applications, this could mean that the libraries the application needs are still present and providing the expected set of features and behaviors. When we talk about Python, it means continuing to provide the modules the Python script requires or the language features it needs.

Red Hat Enterprise Linux 9 was released in May of 2022. At the same time, the Red Hat Enterprise Linux 9: Application Compatibility Guide (RHEL ACG) was published, and the Red Hat Enterprise Linux Container Compatibility Matrix (RHEL CCM) was updated. These two guides are key to helping developers learn about the guidelines they can follow to ensure application compatibility with a given RHEL release.

In the first two parts of this series, we will focus on the RHEL ACG.

Defining API and ABI

This is a quick recap of the detailed definitions in the compatibility guide. An API is the application programming interface, and it represents the set of conventions, features, or behaviors at compile time. An ABI is the application binary interface, and it represents the set of conventions, features, or behaviors at run time.

An API can be a source level function call (e.g.,exit(0)). An ABI can be the actual implementation of the void exit(int status) function in the C library.

Components of the application compatibility guide

The ACG is split into two major sections:

• Guidelines for preserving application compatibility across minor and major RHEL versions.
• The binary rpm package list and the compatibility guarantees.

The distribution places the binary rpm packages into one of four compatibility levels. It can be viewed as a narrowing set of compatibility guarantees across minor and major upgrade paths.

Following the guidelines and using only packages that provide the guarantees your application needs will ensure that your application remains as compatible as possible across minor and major RHEL version upgrades.

To be compatible with minor version upgrades, it requires you to use only packages in compatibility level 2 or level 1, with review of packages used in level 3. To be compatible with major version upgrades, it requires you to use only packages in compatibility level 1, with review of packages used in level 3.

The default for packages in RHEL is compatibility level 2, which ensures that applications keep working across minor version upgrades.

Workloads, services, containers, and packages

You’ll notice that we talk a lot about packages. We talk about packages because it allows developers to decide which parts of decomposed workloads will be compatible with RHEL for a long time and which parts you might have to recompile, rewrite, or forward port.

At the highest level, you are going to have a workload that you want to support over time. That workload does something useful. Workloads can be managed with automation like Ansible or Backstage. I am not going to talk about workloads because as an abstraction, they are useful for talking at a very high level. When you think about it, the workload of “transactional request processing system” is too abstract for us to talk about compatibility.

A workload can be decomposed into services, and at this point it starts getting closer to the level at which we are talking about cross-RHEL-release compatibility guidelines. Services have concrete instantiations like an AMQP (Advanced Message Queuing Protocol) server running locally that handles messages (e.g., RabbitMQ).

When it comes to the smallest installable unit of something on RHEL, we can talk generally about containers or rpm packages. I’m going to defer the conversation about containers until part 3 of this series since the compatibility of containers is covered in the Red Hat Enterprise Linux Container Compatibility Matrix.

So either we are talking about the rpm packages in the container, or we’re talking about rpm packages on the host. As a developer you are still responsible for the decomposition of the workload and services into packages (software collections or modules are still delivered as packages). If packages change between major version upgrades then RHEL Leapp is there to help ensure the same features are present even if the package changes names.

Example: A RHEL 7 application written in C

Explaining application compatibility guidelines is easier with an example. Say you are building an application in C that you started developing on RHEL 7, and you are now looking at RHEL 8 and RHEL 9 for eventual deployment.

Let’s dive in by using a simple C “Hello World” example and see what the application compatibility guidelines say about each of the development steps in building the application.

#include <stdio.h>

int
main (void)
{
  printf ("Hello World!\n");
  return 0;
}

Compile, inspect, and run as follows:

$ gcc -o helloworld helloworld.c

$ ldd helloworld
    linux-vdso.so.1 =>  (0x00007ffd09d75000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fe51a633000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe51aa01000)

$ readelf -W --dyn-syms helloworld

Symbol table '.dynsym' contains 4 entries:
   Num: Value      Size Type Bind   Vis  Ndx Name
  0: 0000000000000000 0 NOTYPE  LOCAL  DEFAULT  UND
  1: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND puts@GLIBC_2.2.5 (2)
  2: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
  3: 0000000000000000 0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__

$./helloworld
Hello World!

There are several important takeaways from this example such as:

• Links only the libraries it needs (ClLibrary: glibc, implicitly).
  • Improves compatibility with future RHEL versions by limiting library use.
• Uses C library header files for the APIs it uses (i.e., #include directive).
  • Ensures the development packages are installed provide those files and ensures any compatibility mechanisms are active.
• Application is executed in an environment that is as new as the system it was compiled on.
  • Backwards compatibility is guaranteed.
• Does not use static linking.
  • Robust runtime behavior of statically linked applications requires that the runtime environment match exactly the build time environment.
• Avoids explicit dependency on a given Linux kernel version.
  • Improves compatibility if the application is later packaged as a container.
  • Improves compatibility with future RHEL kernel versions.

The RHEL7 Application Compatibility Guide recommends all these points and more.

So you might be asking, “Great, but what does that mean for my RHEL 8 and RHEL 9 migration?” Let's look first at the dependencies of the application. In this case, it’s only two dynamic symbols in the rpm package glibc, i.e. puts@GLIBC_2.2.5 and __libc_start_main@GLIBC_2.2.5. Tracking which package provides these symbols is done by rpm and dynamic shared object names, and symbol set provides. It is a topic for another article, so for now, we'll skip this part.

The glibc binary rpm package is one of a small set of packages that is in compatibility level 1, and these are very important packages that are guaranteed to have a compatible ABI for at least three major RHEL releases. That means that the application we just created in this example should run correctly in RHEL 7, RHEL 8, and RHEL 9. Let's try it out.

On RHEL 7:

$ uname -r
3.10.0-1160.88.1.el7.x86_64

$ sha1sum helloworld
beb000e63f609b09583a719ece01ea58b50dd2f8  helloworld

$./helloworld
Hello World!

On RHEL 8:

$ uname -r
4.18.0-468.el8.x86_64

$ sha1sum helloworld
beb000e63f609b09583a719ece01ea58b50dd2f8  helloworld

$./helloworld
Hello World!

On RHEL 9:

$ uname -r
5.14.0-162.21.1.el9_1.x86_64

$ sha1sum helloworld
beb000e63f609b09583a719ece01ea58b50dd2f8  helloworld

$./helloworld
Hello World!

Having an application that only depends on compatibility level 1 packages allows the binary to run across three major RHEL releases without any issues. By 2024, that will be ten years of runtime compatibility!

Example: A RHEL 7 application written in C using OpenSSL

Let us take that example a bit further and try to use a library like OpenSSL that is only compatibility level 2, which means the ABI guarantee is only for the minor version upgrades of RHEL. That means that if you compile on RHEL 7.0, the application should still be working in RHEL 7.9 (last y-stream release), but is not guaranteed to work in RHEL 8.0 without recompilation in that distribution.

The example program here uses OpenSSL’s BIO interface to read from standard input and write the same thing to standard output. While this doesn’t exercise all of OpenSSL’s features, it does help us illustrate application compatibility.

#include <openssl/ssl.h>
#include <openssl/bio.h>
#include <stdlib.h>
#define BUFSIZE 4096

int
main(void)
{
  char buf[BUFSIZE];
  int bin, bout;
  BIO *bio_stdin, *bio_stdout;
  bio_stdin  = BIO_new_fp(stdin,  BIO_NOCLOSE);
  bio_stdout = BIO_new_fp(stdout, BIO_NOCLOSE);
  if (bio_stdin == NULL || bio_stdout == NULL)
    exit (1);
  while ((bin = BIO_read (bio_stdin, buf, BUFSIZE)) > 0)
   {
     bout = BIO_write (bio_stdout, buf, bin);
     if (bin != bout)
       exit (1);
   }
  BIO_free (bio_stdout);
  BIO_free (bio_stdin);
  return 0;
}

Compiled, inspected, and run like the following:

$ gcc -o bio-cp bio-cp.c -lssl -lcrypto

$ ldd./bio-cp
    linux-vdso.so.1 =>  (0x00007ffc4cfa1000)
    libssl.so.10 => /lib64/libssl.so.10 (0x00007f1811c65000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f1811802000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f1811434000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f18111e7000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f1810efe000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f1810cfa000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f1810ac7000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f18108c3000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f18106ad000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1811ed7000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f181049d000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f1810299000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f181007f000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f180fe63000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f180fc3c000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f180f9da000)

$ readelf -W --dyn-syms bio-cp

Symbol table '.dynsym' contains 15 entries:
   Num: Value      Size Type Bind   Vis  Ndx Name
  0: 0000000000000000 0 NOTYPE  LOCAL  DEFAULT  UND
  1: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND BIO_write@libcrypto.so.10 (3)
  2: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND BIO_read@libcrypto.so.10 (3)
  3: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND BIO_free@libcrypto.so.10 (3)
  4: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND exit@GLIBC_2.2.5 (2)
  5: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND BIO_new_fp@libcrypto.so.10 (3)
  6: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
  7: 0000000000000000 0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
  8: 0000000000601060 8 OBJECT  GLOBAL DEFAULT   25 stdout@GLIBC_2.2.5 (2)
  9: 0000000000601054 0 NOTYPE  GLOBAL DEFAULT   24 _edata
10: 0000000000601078 0 NOTYPE  GLOBAL DEFAULT   25 _end
11: 0000000000601068 8 OBJECT  GLOBAL DEFAULT   25 stdin@GLIBC_2.2.5 (2)
12: 0000000000400640 0 FUNC GLOBAL DEFAULT   11 _init
13: 0000000000601054 0 NOTYPE  GLOBAL DEFAULT   25 __bss_start
14: 0000000000400914 0 FUNC GLOBAL DEFAULT   14 _fini

$./bio-cp < helloworld.c
#include <stdio.h>

int
main (void)
{
  printf ("Hello World!\n");
  return 0;
}

$ echo $?
0

The application compatibility guidelines for RHEL 7 state that to be compatible, you must recompile at each major release to ensure ABI compatibility. Again, this is because OpenSSL is in compatibility level 2. Let's put this to the test.

On RHEL 8:

$./bio-cp
./bio-cp: error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory

On RHEL 9:

$./bio-cp
./bio-cp: error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory

The default version of OpenSSL in RHEL7 is 1.0. In RHEL8 it is 1.1.1, and in RHEL9 it is 3.0. Each version is unique, and to use them requires the application to be compiled against the specific version in the distribution.

The application compatibility guide lists OpenSSL as compatibility level 2, and it bears repeating that such libraries have guaranteed ABI compatibility only within the major version of RHEL in which they were released.

The recent migration of RHEL9 to OpenSSL 3.0 was a technically complex migration. Red Hat went above and beyond by providing customers with compatibility packages to facilitate application developers. These compatibility packages provide the libraries required to meet the ABI requirements of OpenSSL using applications. Similar packages were also provided in RHEL 8 to enable the migration from OpenSSL 1.0 to 1.1.1. Lets try out the compatibility packages.

On RHEL 8 with compat-openssl10:

$ dnf install compat-openssl10
...

$ ldd bio-cp
    linux-vdso.so.1 (0x00007fff5df5b000)
    libssl.so.10 => /lib64/libssl.so.10 (0x00007fac6d7ed000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fac6d38b000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fac6cfc6000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fac6cdc2000)
    libz.so.1 => /lib64/libz.so.1 (0x00007fac6cbaa000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fac6da5c000) 

$./bio-cp < helloworld.c
#include <stdio.h>

int
main (void)
{
  printf ("Hello World!\n");
  return 0;
}

While the ACG says not to rely on such compatibility for compatibility level 2 packages, there are cases like this one with OpenSSL where Red Hat has gone above and beyond to ensure customer success.

Learn more about RHEL ACG

We have discussed application compatibility, and how Red Hat provides guidelines and guarantees to help developers create applications that are compatible with minor and major version upgrades for Red Hat Enterprise Linux. We have looked at the first important document that provides those guidelines for RHEL 9, the Red Hat Enterprise Linux 9: Application Compatibility Guide (RHEL ACG). We have looked at two examples that showcase application compatibility across major version upgrades.

I encourage all developers out there to read the Red Hat Enterprise Linux 9: Application Compatibility Guide (RHEL ACG), and make sure you are following the best practice guidelines to meet the compatibility needs of your application, service, or workload. Stay tuned for part 2 of this series where we will look again at the RHEL ACG, but dive into more complex examples. Part 3 will cover container compatibility and describe the Red Hat Enterprise Linux Container Compatibility Matrix (RHEL CCM).