Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

How to use RHEL application compatibility guidelines

May 18, 2023
Carlos O'Donell
Related topics:
C, C#, C++Linux
Related products:
Red Hat Enterprise Linux

Share:

    In this three-part series, we explore the Red Hat Enterprise Linux published application compatibility guidelines (ACG), and how developers can use them to ensure their application remains compatible with future releases of RHEL. Building applications can be difficult, and building applications that continue to operate after an in-place distribution upgrade is even harder. How does Red Hat Enterprise Linux make it easier? It provides guidelines and guarantees that you can follow to improve application compatibility.

    In this article, we will expand on the concept of application compatibility. In the second part, we will review the topic of compatibility with more examples. In the third article, we will discuss container userspace compatibility with the host kernel services.

    What is application compatibility?

    What we call application compatibility is traditionally referred to as backwards compatibility. It is the ability to run an unmodified application binary on the current or newer version of the distribution and have it operate correctly (i.e., compatible with the release of the distribution).

    Maintaining compatibility

    Application compatibility is maintained by ensuring that the dependencies of the application continue to be provided and that the application continues to function as intended.

    When we talk about C or C++ applications, this could mean that the libraries the application needs are still present and providing the expected set of features and behaviors. When we talk about Python, it means continuing to provide the modules the Python script requires or the language features it needs.

    Red Hat Enterprise Linux 9 was released in May of 2022. At the same time, the Red Hat Enterprise Linux 9: Application Compatibility Guide (RHEL ACG) was published, and the Red Hat Enterprise Linux Container Compatibility Matrix (RHEL CCM) was updated. These two guides are key to helping developers learn about the guidelines they can follow to ensure application compatibility with a given RHEL release.

    In the first two parts of this series, we will focus on the RHEL ACG.

    Defining API and ABI

    This is a quick recap of the detailed definitions in the compatibility guide. An API is the application programming interface, and it represents the set of conventions, features, or behaviors at compile time. An ABI is the application binary interface, and it represents the set of conventions, features, or behaviors at run time.

    An API can be a source level function call (e.g.,exit(0)). An ABI can be the actual implementation of the void exit(int status) function in the C library.

    Components of the application compatibility guide

    The ACG is split into two major sections:

    • Guidelines for preserving application compatibility across minor and major RHEL versions.
    • The binary rpm package list and the compatibility guarantees.

    The distribution places the binary rpm packages into one of four compatibility levels. It can be viewed as a narrowing set of compatibility guarantees across minor and major upgrade paths.

    ⇓ Compatibility Level / RHEL Version ⇒

    RHEL X.Y

    RHEL X.Y+1

    RHEL X+1

    RHEL X+2

    1

    API and ABI compatible

    API and ABI compatible

    ABI compatible

    ABI compatible

    2

    API and ABI compatible

    API and ABI compatible

       

    3

    API and ABI defined by life cycle

         

    4

    API and ABI subject to change at any time

         

    Following the guidelines and using only packages that provide the guarantees your application needs will ensure that your application remains as compatible as possible across minor and major RHEL version upgrades.

    To be compatible with minor version upgrades, it requires you to use only packages in compatibility level 2 or level 1, with review of packages used in level 3. To be compatible with major version upgrades, it requires you to use only packages in compatibility level 1, with review of packages used in level 3.

    The default for packages in RHEL is compatibility level 2, which ensures that applications keep working across minor version upgrades.

    Workloads, services, containers, and packages

    You’ll notice that we talk a lot about packages. We talk about packages because it allows developers to decide which parts of decomposed workloads will be compatible with RHEL for a long time and which parts you might have to recompile, rewrite, or forward port.

    At the highest level, you are going to have a workload that you want to support over time. That workload does something useful. Workloads can be managed with automation like Ansible or Backstage. I am not going to talk about workloads because as an abstraction, they are useful for talking at a very high level. When you think about it, the workload of “transactional request processing system” is too abstract for us to talk about compatibility.

    A workload can be decomposed into services, and at this point it starts getting closer to the level at which we are talking about cross-RHEL-release compatibility guidelines. Services have concrete instantiations like an AMQP (Advanced Message Queuing Protocol) server running locally that handles messages (e.g., RabbitMQ).

    When it comes to the smallest installable unit of something on RHEL, we can talk generally about containers or rpm packages. I’m going to defer the conversation about containers until part 3 of this series since the compatibility of containers is covered in the Red Hat Enterprise Linux Container Compatibility Matrix.

    So either we are talking about the rpm packages in the container, or we’re talking about rpm packages on the host. As a developer you are still responsible for the decomposition of the workload and services into packages (software collections or modules are still delivered as packages). If packages change between major version upgrades then RHEL Leapp is there to help ensure the same features are present even if the package changes names.

    Example: A RHEL 7 application written in C

    Explaining application compatibility guidelines is easier with an example. Say you are building an application in C that you started developing on RHEL 7, and you are now looking at RHEL 8 and RHEL 9 for eventual deployment.

    Let’s dive in by using a simple C “Hello World” example and see what the application compatibility guidelines say about each of the development steps in building the application.

    #include <stdio.h>
    
    int
    main (void)
    {
      printf ("Hello World!\n");
      return 0;
    }

    Compile, inspect, and run as follows:

    $ gcc -o helloworld helloworld.c
    
    $ ldd helloworld
        linux-vdso.so.1 =>  (0x00007ffd09d75000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe51a633000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe51aa01000)
    
    $ readelf -W --dyn-syms helloworld
    
    Symbol table '.dynsym' contains 4 entries:
       Num: Value      Size Type Bind   Vis  Ndx Name
      0: 0000000000000000 0 NOTYPE  LOCAL  DEFAULT  UND
      1: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND puts@GLIBC_2.2.5 (2)
      2: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
      3: 0000000000000000 0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
    
    $./helloworld
    Hello World!

    There are several important takeaways from this example such as:

    • Links only the libraries it needs (ClLibrary: glibc, implicitly).
      • Improves compatibility with future RHEL versions by limiting library use.
    • Uses C library header files for the APIs it uses (i.e., #include directive).
      • Ensures the development packages are installed provide those files and ensures any compatibility mechanisms are active.
    • Application is executed in an environment that is as new as the system it was compiled on.
      • Backwards compatibility is guaranteed.
    • Does not use static linking.
      • Robust runtime behavior of statically linked applications requires that the runtime environment match exactly the build time environment.
    • Avoids explicit dependency on a given Linux kernel version.
      • Improves compatibility if the application is later packaged as a container.
      • Improves compatibility with future RHEL kernel versions.

    The RHEL7 Application Compatibility Guide recommends all these points and more.

    So you might be asking, “Great, but what does that mean for my RHEL 8 and RHEL 9 migration?” Let's look first at the dependencies of the application. In this case, it’s only two dynamic symbols in the rpm package glibc, i.e. puts@GLIBC_2.2.5 and __libc_start_main@GLIBC_2.2.5. Tracking which package provides these symbols is done by rpm and dynamic shared object names, and symbol set provides. It is a topic for another article, so for now, we'll skip this part.

    The glibc binary rpm package is one of a small set of packages that is in compatibility level 1, and these are very important packages that are guaranteed to have a compatible ABI for at least three major RHEL releases. That means that the application we just created in this example should run correctly in RHEL 7, RHEL 8, and RHEL 9. Let's try it out.

    On RHEL 7:

    $ uname -r
    3.10.0-1160.88.1.el7.x86_64
    
    $ sha1sum helloworld
    beb000e63f609b09583a719ece01ea58b50dd2f8  helloworld
    
    $./helloworld
    Hello World!

    On RHEL 8:

    $ uname -r
    4.18.0-468.el8.x86_64
    
    $ sha1sum helloworld
    beb000e63f609b09583a719ece01ea58b50dd2f8  helloworld
    
    $./helloworld
    Hello World!

    On RHEL 9:

    $ uname -r
    5.14.0-162.21.1.el9_1.x86_64
    
    $ sha1sum helloworld
    beb000e63f609b09583a719ece01ea58b50dd2f8  helloworld
    
    $./helloworld
    Hello World!
    
    

    Having an application that only depends on compatibility level 1 packages allows the binary to run across three major RHEL releases without any issues. By 2024, that will be ten years of runtime compatibility!

    Example: A RHEL 7 application written in C using OpenSSL

    Let us take that example a bit further and try to use a library like OpenSSL that is only compatibility level 2, which means the ABI guarantee is only for the minor version upgrades of RHEL. That means that if you compile on RHEL 7.0, the application should still be working in RHEL 7.9 (last y-stream release), but is not guaranteed to work in RHEL 8.0 without recompilation in that distribution.

    The example program here uses OpenSSL’s BIO interface to read from standard input and write the same thing to standard output. While this doesn’t exercise all of OpenSSL’s features, it does help us illustrate application compatibility.

    #include <openssl/ssl.h>
    #include <openssl/bio.h>
    #include <stdlib.h>
    #define BUFSIZE 4096
    
    int
    main(void)
    {
      char buf[BUFSIZE];
      int bin, bout;
      BIO *bio_stdin, *bio_stdout;
      bio_stdin  = BIO_new_fp(stdin,  BIO_NOCLOSE);
      bio_stdout = BIO_new_fp(stdout, BIO_NOCLOSE);
      if (bio_stdin == NULL || bio_stdout == NULL)
        exit (1);
      while ((bin = BIO_read (bio_stdin, buf, BUFSIZE)) > 0)
       {
         bout = BIO_write (bio_stdout, buf, bin);
         if (bin != bout)
           exit (1);
       }
      BIO_free (bio_stdout);
      BIO_free (bio_stdin);
      return 0;
    }
    
    

    Compiled, inspected, and run like the following:

    $ gcc -o bio-cp bio-cp.c -lssl -lcrypto
    
    $ ldd./bio-cp
        linux-vdso.so.1 =>  (0x00007ffc4cfa1000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007f1811c65000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f1811802000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f1811434000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f18111e7000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f1810efe000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f1810cfa000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f1810ac7000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f18108c3000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f18106ad000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1811ed7000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f181049d000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f1810299000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f181007f000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f180fe63000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f180fc3c000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f180f9da000)
    
    $ readelf -W --dyn-syms bio-cp
    
    Symbol table '.dynsym' contains 15 entries:
       Num: Value      Size Type Bind   Vis  Ndx Name
      0: 0000000000000000 0 NOTYPE  LOCAL  DEFAULT  UND
      1: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND BIO_write@libcrypto.so.10 (3)
      2: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND BIO_read@libcrypto.so.10 (3)
      3: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND BIO_free@libcrypto.so.10 (3)
      4: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND exit@GLIBC_2.2.5 (2)
      5: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND BIO_new_fp@libcrypto.so.10 (3)
      6: 0000000000000000 0 FUNC GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
      7: 0000000000000000 0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
      8: 0000000000601060 8 OBJECT  GLOBAL DEFAULT   25 stdout@GLIBC_2.2.5 (2)
      9: 0000000000601054 0 NOTYPE  GLOBAL DEFAULT   24 _edata
    10: 0000000000601078 0 NOTYPE  GLOBAL DEFAULT   25 _end
    11: 0000000000601068 8 OBJECT  GLOBAL DEFAULT   25 stdin@GLIBC_2.2.5 (2)
    12: 0000000000400640 0 FUNC GLOBAL DEFAULT   11 _init
    13: 0000000000601054 0 NOTYPE  GLOBAL DEFAULT   25 __bss_start
    14: 0000000000400914 0 FUNC GLOBAL DEFAULT   14 _fini
    
    $./bio-cp < helloworld.c
    #include <stdio.h>
    
    int
    main (void)
    {
      printf ("Hello World!\n");
      return 0;
    }
    
    $ echo $?
    0

    The application compatibility guidelines for RHEL 7 state that to be compatible, you must recompile at each major release to ensure ABI compatibility. Again, this is because OpenSSL is in compatibility level 2. Let's put this to the test.

    On RHEL 8:

    $./bio-cp
    ./bio-cp: error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory

    On RHEL 9:

    $./bio-cp
    ./bio-cp: error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory

    The default version of OpenSSL in RHEL7 is 1.0. In RHEL8 it is 1.1.1, and in RHEL9 it is 3.0. Each version is unique, and to use them requires the application to be compiled against the specific version in the distribution.

    The application compatibility guide lists OpenSSL as compatibility level 2, and it bears repeating that such libraries have guaranteed ABI compatibility only within the major version of RHEL in which they were released.

    The recent migration of RHEL9 to OpenSSL 3.0 was a technically complex migration. Red Hat went above and beyond by providing customers with compatibility packages to facilitate application developers. These compatibility packages provide the libraries required to meet the ABI requirements of OpenSSL using applications. Similar packages were also provided in RHEL 8 to enable the migration from OpenSSL 1.0 to 1.1.1. Lets try out the compatibility packages.

    On RHEL 8 with compat-openssl10:

    $ dnf install compat-openssl10
    ...
    
    $ ldd bio-cp
        linux-vdso.so.1 (0x00007fff5df5b000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007fac6d7ed000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fac6d38b000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fac6cfc6000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fac6cdc2000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fac6cbaa000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fac6da5c000) 
    
    $./bio-cp < helloworld.c
    #include <stdio.h>
    
    int
    main (void)
    {
      printf ("Hello World!\n");
      return 0;
    }

    While the ACG says not to rely on such compatibility for compatibility level 2 packages, there are cases like this one with OpenSSL where Red Hat has gone above and beyond to ensure customer success.

    Learn more about RHEL ACG

    We have discussed application compatibility, and how Red Hat provides guidelines and guarantees to help developers create applications that are compatible with minor and major version upgrades for Red Hat Enterprise Linux. We have looked at the first important document that provides those guidelines for RHEL 9, the Red Hat Enterprise Linux 9: Application Compatibility Guide (RHEL ACG). We have looked at two examples that showcase application compatibility across major version upgrades.

    I encourage all developers out there to read the Red Hat Enterprise Linux 9: Application Compatibility Guide (RHEL ACG), and make sure you are following the best practice guidelines to meet the compatibility needs of your application, service, or workload. Stay tuned for part 2 of this series where we will look again at the RHEL ACG, but dive into more complex examples. Part 3 will cover container compatibility and describe the Red Hat Enterprise Linux Container Compatibility Matrix (RHEL CCM).

    Last updated: August 14, 2023

    Related Posts

    • How to install multiple versions of Python on Red Hat Enterprise Linux

    • What's new in Red Hat Enterprise Linux 9.1

    • Build your first application using Python 3.5 on RHEL 7 with containers and Red Hat Software Collections

    • Python wheels, AI/ML, and ABI compatibility

    • 5 principles for deploying your API from a CI/CD pipeline

    Recent Posts

    • More Essential AI tutorials for Node.js Developers

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    What’s up next?

    Advanced Linux Commands tile card - updated

    Download the Advanced Linux Commands Cheat Sheet, which presents a collection of Linux commands and executables for developers who are using the Linux operating system in advanced programming scenarios.

    Get the cheat sheet
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue