Red Hat Enterprise Linux (RHEL), in version 8, introduced modules as a higher-level concept for packaging software stacks. Modules enable new features such as adding alternative versions of stacks, called streams. That's great, but what if you want to patch a stream? Is it possible? It is. Is it more difficult than patching non-modular software? Slightly. This article shows you how to patch a module stream while avoiding the invisible package problem.
Patching a module in RHEL
Red Hat Enterprise Linux is open source. That means you can take the code sources, change them, recompile them, and use or redistribute the modified software. As an example, we can change the HTTPD web server to report a different server name in the HTTP response headers.
To get started, install an httpd RPM package, start the HTTPD server, and check the server name. I have highlighted the relevant lines from the output in bold:
# yum install httpd
Last metadata expiration check: 0:03:40 ago on Fri 16 Jul 2021 12:51:49 PM CEST.
Dependencies resolved.
==========================================================================================
Package Arch Version Repository Size
==========================================================================================
Installing:
httpd x86_64 2.4.37-40.module+el8.5.0+11022+1c90597b rhel-8.5.0-appstream 1.4 M
Installing dependencies:
httpd-filesystem
noarch 2.4.37-40.module+el8.5.0+11022+1c90597b rhel-8.5.0-appstream 39 k
httpd-tools x86_64 2.4.37-40.module+el8.5.0+11022+1c90597b rhel-8.5.0-appstream 106 k
mod_http2 x86_64 1.15.7-3.module+el8.4.0+8625+d397f3da pulp-appstream 154 k
redhat-logos-httpd
noarch 84.5-1.el8 rhel-8.5.0-baseos 29 k
Enabling module streams:
httpd 2.4
Transaction Summary
==========================================================================================
Install 5 Packages
Total download size: 1.7 M
Installed size: 4.9 M
Is this ok [y/N]: y
[…]
Complete!
# systemctl start httpd
$ wget --no-proxy -S -O /dev/null http://localhost/
--2021-07-16 12:58:54-- http://localhost/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 403 Forbidden
Date: Fri, 16 Jul 2021 10:58:54 GMT
Server: Apache/2.4.37 (Red Hat Enterprise Linux)
Last-Modified: Mon, 12 Jul 2021 19:36:32 GMT
ETag: "133f-5c6f23d09f000"
Accept-Ranges: bytes
Content-Length: 4927
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
2021-07-16 12:58:54 ERROR 403: Forbidden.
The output shows that the httpd-2.4.37-40.module+el8.5.0+11022+1c90597b RPM package was installed from the httpd:2.4 module stream and that the server reports Apache/2.4.37 (Red Hat Enterprise Linux).
Our quest is to patch the module to report My Linux instead.
Step 1: Build a new package
First, obtain the source RPM package, httpd-2.4.37-40.module+el8.5.0+11022+1c90597b.src.rpm, which corresponds to our example. Unpack it and apply the following patch to a specification file, as explained in the Red Hat documentation:
--- a/httpd.spec
+++ b/httpd.spec
@@ -13,7 +13,7 @@
Summary: Apache HTTP Server
Name: httpd
Version: 2.4.37
-Release: 40%{?dist}
+Release: 41%{?dist}
URL: https://httpd.apache.org/
Source0: https://www.apache.org/dist/httpd/httpd-%{version}.tar.bz2
Source2: httpd.logrotate
@@ -370,7 +370,7 @@ interface for storing and accessing per-user session data.
%patch211 -p1 -b .CVE-2020-11984
# Patch in the vendor string
-sed -i '/^#define PLATFORM/s/Unix/%{vstring}/' os/unix/os.h
+sed -i '/^#define PLATFORM/s/Unix/My Linux/' os/unix/os.h
sed -i 's/@RELEASE@/%{release}/' server/core.c
# Prevent use of setcap in "install-suexec-caps" target.
@@ -870,6 +870,9 @@ rm -rf $RPM_BUILD_ROOT
%{_rpmconfigdir}/macros.d/macros.httpd
%changelog
+* Wed Jun 23 2021 Petr Pisar <ppisar@redhat.com> - 2.4.37-41
+- Modified server platform
+
* Fri May 14 2021 Lubos Uhliarik <luhliari@redhat.com> - 2.4.37-40
- Resolves: #1952557 - mod_proxy_wstunnel.html is a malformed XML
- Resolves: #1937334 - SSLProtocol with based virtual hosts
Now, build the modified package with a rpmbuild tool. This results in the following binary packages:
$ ls
httpd-2.4.37-41.el8.x86_64.rpm mod_ldap-2.4.37-41.el8.x86_64.rpm
httpd-debuginfo-2.4.37-41.el8.x86_64.rpm mod_ldap-debuginfo-2.4.37-41.el8.x86_64.rpm
httpd-debugsource-2.4.37-41.el8.x86_64.rpm mod_proxy_html-2.4.37-41.el8.x86_64.rpm
httpd-devel-2.4.37-41.el8.x86_64.rpm mod_proxy_html-debuginfo-2.4.37-41.el8.x86_64.rpm
httpd-filesystem-2.4.37-41.el8.noarch.rpm mod_session-2.4.37-41.el8.x86_64.rpm
httpd-manual-2.4.37-41.el8.noarch.rpm mod_session-debuginfo-2.4.37-41.el8.x86_64.rpm
httpd-tools-2.4.37-41.el8.x86_64.rpm mod_ssl-2.4.37-41.el8.x86_64.rpm
httpd-tools-debuginfo-2.4.37-41.el8.x86_64.rpm mod_ssl-debuginfo-2.4.37-41.el8.x86_64.rpm
Step 2: Create a nonmodular repository
Next, turn the directory into a YUM repository. Let's say that the repository is located in the working directory /root/repos/myhttpd, so all write operations there must be performed by a superuser:
# createrepo_c .
Directory walk started
Directory walk done - 16 packages
Temporary output repo path: ./.repodata/
Preparing sqlite DBs
Pool started (with 5 workers)
Pool finished
Register the repository to YUM under the name myhttpd by creating the /etc/yum.repos.d/devel.repo file with the following content:
[myhttpd]
name=myhttpd packages
baseurl=file:///root/repos/myhttpd/
enabled=1
gpgcheck=0
The invisible package problem
Next, let's try to update the system to install the patched package:
# yum upgrade
myhttpd packages 2.9 MB/s | 3.0 kB 00:00
myhttpd packages 2.5 MB/s | 25 kB 00:00
Dependencies resolved.
Nothing to do.
Complete!
It doesn't work! YUM cannot see your new httpd-2.4.37-41.el8.x86_64 package. Check which packages YUM sees:
$ repoquery httpd
Last metadata expiration check: 0:06:19 ago on Fri 16 Jul 2021 01:31:22 PM CEST.
httpd-0:2.4.37-10.module+el8+2764+7127e69e.x86_64
httpd-0:2.4.37-11.module+el8.0.0+2969+90015743.x86_64
httpd-0:2.4.37-12.module+el8.0.0+4096+eb40e6da.x86_64
httpd-0:2.4.37-16.module+el8.1.0+4134+e6bad0ed.x86_64
httpd-0:2.4.37-21.module+el8.2.0+5008+cca404a3.x86_64
httpd-0:2.4.37-30.module+el8.3.0+7001+0766b9e7.x86_64
httpd-0:2.4.37-39.module+el8.4.0+9658+b87b2deb.x86_64
httpd-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
The new package isn't there, but why? Because packages belonging to an active module stream take precedence over other packages of the same name. This issue is sometimes known as the invisible package problem. We'll explore it in the next section.
Theory of modules
To resolve the invisible package problem, you need to understand how modules work.
Modules are organized into streams (examples include httpd:2.4, perl:5.24, and perl:5.30; also see the yum module list command output). Each stream consists of a series of module versions (such as httpd:2.4:8040020210127115317 and httpd:2.4:8050020210517115912) and each module version lists RPM packages belonging to it (see the Artifacts section in the output from yum module info httpd:2.4).
A module stream is active if the developer enables it explicitly, or if it is the default and has not been explicitly disabled. All packages belonging to the active stream are visible to YUM. All other packages of the same name, including packages not belonging to any module, are invisible.
Correspondingly, when a module stream is not active (is disabled or is nondefault), its packages are invisible, while nonmodular packages with the same name are kept visible.
In a typical Red Hat Enterprise Linux distribution, you can observe changes in visibility as follows:
- Enable the
perl:5.24stream by runningyum enable perl:5.24. - List the Perl packages in the repository by entering
repoquery perl. - Reset the stream through
yum module reset perl. - Enable
perl:5.30in a similar way. - List the packages again and view the differences from the previous listing.
- Reset the stream to
perl:5.26, or whatever the default was on your system. - List the packages again.
After each change, YUM lists different perl packages.
Solving the invisible package problem
In our example, the new httpd-2.4.37-41.el8.x86_64 is currently prevented from being visible. The httpd:2.4 module stream is active and lists an httpd package:
$ yum module info httpd:2.4
[…]
Name : httpd
Stream : 2.4 [d][e][a]
Version : 8050020210517115912
Context : b4937e53
Architecture : x86_64
Profiles : common [d], devel, minimal
Default profiles : common
Repo : rhel-8.5.0-appstream
Summary : Apache HTTP Server
Description : Apache httpd is a powerful, efficient, and extensible HTTP server.
Requires : platform:[el8]
Artifacts : httpd-0:2.4.37-40.module+el8.5.0+11022+1c90597b.src
: httpd-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
[…]
To patch the module, you need to define a new httpd:2.4 module version, list the new httpd-2.4.37-41.el8.x86_64 package there, and add the new module version definition to the repository. After you've done that, YUM will recognize the new package as belonging to the httpd:2.4 stream, and you can continue to the next step.
Step 3: Make the repository modular
Now comes a step specific to modules: Changing a nonmodular repository into a modular one. Copy the httpd:2.4:8050020210517115912:b4937e53:x86_64 module definition from the original repository to the modules.yaml file in the directory with the new package:
# zcat /var/cache/dnf/rhel-8.5.0-appstream-801b3acbf7fb96cf/repodata/7642b0bd7a55141335285144eb537352c85f336de8187ad14aa40b0dbf532463-modules.yaml.gz > modules.yaml
I took the file from a local YUM cache. But you can also find files with names matching *-modules.yaml* on repository mirrors.
Now, locate the module build definition inside the file and delete everything else:
---
document: modulemd
version: 2
data:
name: httpd
stream: "2.4"
version: 8050020210517115912
context: b4937e53
arch: x86_64
[…]
artifacts:
rpms:
- httpd-0:2.4.37-40.module+el8.5.0+11022+1c90597b.src
- httpd-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- httpd-debuginfo-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- httpd-debugsource-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- httpd-devel-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- httpd-filesystem-0:2.4.37-40.module+el8.5.0+11022+1c90597b.noarch
- httpd-manual-0:2.4.37-40.module+el8.5.0+11022+1c90597b.noarch
- httpd-tools-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- httpd-tools-debuginfo-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- mod_http2-0:1.15.7-3.module+el8.4.0+8625+d397f3da.src
- mod_http2-0:1.15.7-3.module+el8.4.0+8625+d397f3da.x86_64
- mod_http2-debuginfo-0:1.15.7-3.module+el8.4.0+8625+d397f3da.x86_64
- mod_http2-debugsource-0:1.15.7-3.module+el8.4.0+8625+d397f3da.x86_64
- mod_ldap-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- mod_ldap-debuginfo-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- mod_md-1:2.0.8-8.module+el8.3.0+6814+67d1e611.src
- mod_md-1:2.0.8-8.module+el8.3.0+6814+67d1e611.x86_64
- mod_md-debuginfo-1:2.0.8-8.module+el8.3.0+6814+67d1e611.x86_64
- mod_md-debugsource-1:2.0.8-8.module+el8.3.0+6814+67d1e611.x86_64
- mod_proxy_html-1:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- mod_proxy_html-debuginfo-1:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- mod_session-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- mod_session-debuginfo-0:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- mod_ssl-1:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
- mod_ssl-debuginfo-1:2.4.37-40.module+el8.5.0+11022+1c90597b.x86_64
...
Update the RPM package list in the /data/artifacts/rpms YAML node to match your new RPM builds:
# sed -i -e 's/-40\.module+el8\.5\.0+11022+1c90597b\./-41.el8./' modules.yaml
Increment the module build version; for example, from 8050020210517115912 to 8050020210517115913:
---
document: modulemd
version: 2
data:
name: httpd
stream: "2.4"
version: 8050020210517115913
context: b4937e53
arch: x86_64
[...]
artifacts:
rpms:
- httpd-0:2.4.37-41.el8.src
- httpd-0:2.4.37-41.el8.x86_64
- httpd-debuginfo-0:2.4.37-41.el8.x86_64
- httpd-debugsource-0:2.4.37-41.el8.x86_64
- httpd-devel-0:2.4.37-41.el8.x86_64
- httpd-filesystem-0:2.4.37-41.el8.noarch
- httpd-manual-0:2.4.37-41.el8.noarch
- httpd-tools-0:2.4.37-41.el8.x86_64
- httpd-tools-debuginfo-0:2.4.37-41.el8.x86_64
- mod_http2-0:1.15.7-3.module+el8.4.0+8625+d397f3da.src
- mod_http2-0:1.15.7-3.module+el8.4.0+8625+d397f3da.x86_64
- mod_http2-debuginfo-0:1.15.7-3.module+el8.4.0+8625+d397f3da.x86_64
- mod_http2-debugsource-0:1.15.7-3.module+el8.4.0+8625+d397f3da.x86_64
- mod_ldap-0:2.4.37-41.el8.x86_64
- mod_ldap-debuginfo-0:2.4.37-41.el8.x86_64
- mod_md-1:2.0.8-8.module+el8.3.0+6814+67d1e611.src
- mod_md-1:2.0.8-8.module+el8.3.0+6814+67d1e611.x86_64
- mod_md-debuginfo-1:2.0.8-8.module+el8.3.0+6814+67d1e611.x86_64
- mod_md-debugsource-1:2.0.8-8.module+el8.3.0+6814+67d1e611.x86_64
- mod_proxy_html-1:2.4.37-41.el8.x86_64
- mod_proxy_html-debuginfo-1:2.4.37-41.el8.x86_64
- mod_session-0:2.4.37-41.el8.x86_64
- mod_session-debuginfo-0:2.4.37-41.el8.x86_64
- mod_ssl-1:2.4.37-41.el8.x86_64
- mod_ssl-debuginfo-1:2.4.37-41.el8.x86_64
...
If the module lists other packages, you can delete them.
Finally, regenerate the repository metadata so that it picks up the new module definition from the modules.yaml file in the local directory:
# createrepo_c .
Directory walk started
Directory walk done - 16 packages
Temporary output repo path: ./.repodata/
Preparing sqlite DBs
Pool started (with 5 workers)
Pool finished
You can check that the module definition was imported under the known *-module.yaml.* filename:
$ ls repodata/
733d406732770bce66f8b790a92e933559903e7c3360a1bada3de366c130fc7c-other.sqlite.bz2
a99faa19e499bec174b3c3c682d150715fc66015da00511aa9f78691a3670708-other.xml.gz
aaffc762d36b0389d602c9c37db74242cae8d37ea89d3aa3e4ca2b4cc4099b0d-primary.sqlite.bz2
d10db6feb91cc5f185218367162cdbab49780343a82efe23e6d8c0e14f4effcb-filelists.xml.gz
e51d17bf9000bd130f99edd8ff2923977c8c74a0b5829116e36299fb46a440e9-primary.xml.gz
f98a57f75a9fb84f1ce0313ee22e435eebb6134a28f7568ad8b8b4e14be38285-filelists.sqlite.bz2
ff2b17e5a515266023ccc983a8cf12401ae4d2c2049683e76ad09f6b5cea48ba-modules.yaml.gz
repomd.xml
Now you can delete the ./modules.yaml file. You don't need it anymore. The repository is now modular.
Note: Importing modular metadata from a modules.yaml file is a new feature of createrepo-c-0.16.2. If you have an older version, you need to use the modifyrepo_c tool after running createrepo_c.
Step 4: Install the package from the modular repository
We are nearly done. Try updating the system again:
# yum upgrade
myhttpd packages 1.4 MB/s | 26 kB 00:00
Dependencies resolved.
==========================================================================================
Package Architecture Version Repository Size
==========================================================================================
Upgrading:
httpd x86_64 2.4.37-41.el8 myhttpd 1.4 M
httpd-filesystem noarch 2.4.37-41.el8 myhttpd 37 k
httpd-tools x86_64 2.4.37-41.el8 myhttpd 104 k
Transaction Summary
==========================================================================================
Upgrade 3 Packages
Total size: 1.5 M
Is this ok [y/N]: y
And that's it. It works. Hooray!
Note: If YUM did not refresh the repository, it might be because you performed the steps too quickly. Clean up the cache with rm -rf /var/cache/dnf/myhttpd* and try again.
Step 5: Verify the patched package
Finally, you can check the Server header that the server returns:
$ wget --no-proxy -S -O /dev/null http://localhost/
--2021-07-16 15:15:56-- http://localhost/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 403 Forbidden
Date: Fri, 16 Jul 2021 13:15:56 GMT
Server: Apache/2.4.37 (My Linux)
Last-Modified: Mon, 12 Jul 2021 19:36:32 GMT
ETag: "133f-5c6f23d09f000"
Accept-Ranges: bytes
Content-Length: 4927
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
2021-07-16 15:15:56 ERROR 403: Forbidden.
The line Server: Apache/2.4.37 (My Linux) shows that the server is running your patched module.
Versioning patched modules and packages
What version should you use for the patched modules? Currently, it doesn't matter much because YUM merges all module versions of a single stream together. But I recommend incrementing the last digit, as we did in our example. The module version is basically a timestamp. Incrementing the last digit means moving a second ahead. It's improbable that Red Hat would release two module versions with one-second delays. Thus, when Red Hat releases a new version, it's recognized as more recent than your version, and replaces your version.
The version number could matter if you want to change other modular metadata, such as modular dependencies. Then the highest module version wins.
What about the RPM version string? Inside a stream, a standard RPM epoch-version-release comparison is used to update a modular package to another modular package. If Red Hat released a new module update, the httpd package would be called something like httpd-0:2.4.37-41.module+el8.5.0+11022+1c90597b. That's fine because that would be a higher RPM version string than yours and the new update would win:
$ rpmdev-vercmp 0:2.4.37-41.el8 0:2.4.37-41.module+el8.5.0+11022+1c90597b
0:2.4.37-41.el8 < 0:2.4.37-41.module+el8.5.0+11022+1c90597b
If you want your RPM package to win over future Red Hat updates, choose a reasonably high release number. The process is the same as what would you do in a nonmodular scenario.
Special situations and warnings
Sometimes there are multiple modules with the same version but a different context value. What does the context mean? Which context should you use? Can you change it?
The context distinguishes modules that were built from the same sources but for different environments. For instance, the perl-DBI:1.641 modules found in Red Hat Enterprise Linux 8.3 are built three times for three different Perl versions, so there are three different contexts of them. When you patch a module, don't change the context. Copying the old value is the safest approach.
I'll conclude by listing a few shortcuts that are not recommended for patching modules:
- Installing from a local file. You can use
yum upgrade ./httpd-*.rpm, but the results won't last long. A package installed like that won't be recognized as belonging to any module and could be expelled from future YUM transactions, resulting in a dependency conflict on an RPM level. Also, having a package outside a repository makes it difficult to deploy to multiple machines or reinstall the package. - Adding a
module_hotfixes=truestatement to a YUM configuration file for a nonmodular repository. While this technique works as a last resort for overriding any modular content, the hammer is too big for the nail. It does not play nicely when multiple module streams provide the same package, or if all the streams are disabled. - Omitting the zero epoch from an artifacts list in the module definition. Don't do it. YUM won't understand it. If your RPM package has no epoch number, write
0. You can userpm -q --qf '%{NAME}-%{EPOCHNUM}:%{VERSION}-%{RELEASE}.%{ARCH}\n' -p httpd-2.4.37-41.el8.x86_64.rpmto obtain the right value:httpd-0:2.4.37-41.el8.x86_64.
References
I recommend reading the module definition format.
Last updated: September 19, 2022