Thousands of PyPI and RubyGems RPMs now available for RHEL 9

Red Hat Enterprise Linux 9 now offers convenient (but unsupported) access to RPMs from two of the largest and most popular code repositories: The Python Package Index (PyPI) for Python and the RubyGems collection for Ruby. This new offering makes it easier to use thousands of community libraries in your projects. We'll look at the repositories in this article.

The Red Hat repository ecosystem and COPR

Red Hat supports about 2,500 packages. They are of very high quality, created and maintained by experts. Outside our supported offerings for RHEL 9, we also have Extra Packages for Enterprise Linux (EPEL 9) with an additional 3,000 packages. But there are a lot of other important libraries and utilities on the internet.

We certainly cannot package everything. But we have decided to provide you with an additional 151,000 RPM packages by packaging everything we can (with an exception we'll discuss in a moment) in PyPI and RubyGems. The packages have been added to another set of unsupported packages called COPR, which stands for community projects. Both EPEL and COPR were formed by the Fedora project.

Benefits of using the signed Red Hat repositories

As noted, Red Hat does not offer support for COPR packages. Each is provided as-is, without warranty. If you have an issue with some package, please contact an upstream author.

However, Red Hat does provide some security through signatures. Packages on PyPI are not signed, but Red Hat signs the RPM packages in COPR repositories. This means that you can audit the packages: You know which files belong to which package and vice versa, and you will be able to check whether a file was altered by a user.

To enable the PyPI COPR on your computer, enter:

# dnf copr enable @copr/PyPI epel-9-x86_64

Now, if pip install foo works for you, installing python3-foo from this repository will work too.

 To enable the RubyGems COPR, enter:

# dnf copr enable @rubygems/rubygems epel-9-x86_64 

Having enabled the COPRs on your system, you can get the signed keys by entering:

# dnf install distribution-gpg-keys
# rpm --import /usr/share/distribution-gpg-keys/copr/copr-@copr-PyPI.gpg
# rpm --import /usr/share/distribution-gpg-keys/copr/copr-@rubygems-rubygems.gpg

Details and limitations

Not all packages from PyPI and RubyGems could be included in our COPRs. Some packages suffered from build problems, which I'll describe later. Others were excluded due to licensing conflicts. If the maintainers of a package fail to explicitly assign a free or open source license, they leave the gem by default under conventional "All rights reserved" copyright. Because these packages are not open source, we cannot distribute them.

We provide the latest version of each Python package and RubyGem. When a new version is published, we rebuild it and update the repository. The previous version is deleted after 14 days. If you need an older version, upload it to Red Hat Satellite or keep a local copy.

What we built from PyPI

Many improvements to Python packaging were made in recent years, both upstream and downstream in RPM.

With new RPM macros created by Python maintainers in Red Hat, it is now possible to create a deterministic converter from Python package metadata to RPM specfiles. We used a new tool, pyp2spec, making use of those new RPM macros, to rebuild PyPI packages as RPMs in COPR.

Starting in December 2021, we tried to rebuild all packages on PyPI (more than 330,000 at the time) in Fedora Rawhide. When it became possible to build packages for EPEL 9 in COPR, we went ahead and reran the build. We completed this project, building 79,842 packages for Red Hat Enterprise Linux 9 and making them available in the PyPI COPR.

Only the packages that were built successfully for Fedora Rawhide were submitted for the Red Hat Enterprise Linux 9 rebuild. Trying to rebuild the whole PyPI repository would require months, and there was little chance that any given EPEL build would succeed if the Rawhide one didn't.

Based on the build logs of the failed packages, we found the following problems:

• 32% had missing build dependencies in our COPR repository, often because a package required a specific version of a dependency.
• 20% didn't have a license or specified a license that wasn't detected as open source.
• About 13% of packages had an error somewhere in the upstream configuration leading to the build failure. Typically, the problem was due either to missing files in the source archive or to failures to import modules that weren't declared as build dependencies.
• 12% of packages didn't have the source archive uploaded to PyPI, which prevented us from building the RPM. If a package was built successfully for Rawhide but not for EPEL, the typical culprit was missing dependencies.

If you encounter a package that's installable from pip, but is not available in our COPR, you can try contacting the upstream maintainers to discuss loosening version requirements for dependencies, fixing the license declaration, uploading the source archive to PyPI, or resolving other problems as the case may be.

For instructions on how to reproduce the build, see our PyPI COPR repository.

What we built from RubyGems

Support for building RPM packages in COPR directly from RubyGems.org was introduced back in 2016. The procedure uses a tool called gem2rpm to convert gem metadata into an RPM specfile and produce an SRPM package based on it.

Utilizing this feature, we have rebuilt all of RubyGems.org for Fedora Rawhide. In this detailed blog post, you can find more information about the success rate, the size of the packages and their metadata, COPR internals, and takeaways for the RPM toolchain.

Countless performance improvements, and months of building later, we are now announcing that all of RubyGems.org was rebuilt for Red Hat Enterprise Linux 9. The RubyGems project in COPR provides 71,952 packages, which is almost half of the RubyGems.org service. We got 19,635 failures because of unmet dependencies, and around 77,000 gems were skipped and not even attempted to be built because of missing licenses. A full 37% of gems in RubyGems do not specify a license.

If your gem isn't available from the RubyGems COPR, it is most likely due to a missing license. Please resolve such issues with the respective gem owners. The same applies to missing dependencies. If you find a problem with the generated specfiles, please file a new issue for gem2rpm.

Conclusion

Our PyPI and RubyGems projects demonstrate Red Hat's desire to help programmers make the most of free and open source resources. During our months-long efforts, we turned up weaknesses in the source repositories that guide upstream developers to produce more robust packages.