How to add packages to Python 2.7 Software Collection

As Software Collections are getting popular, there are more and more people asking how they can build their own collections and/or extend collections in RHSCL. In this article, I will demonstrate how to extend python27 collection from RHSCL 1.2, adding a simple Python extension library. (Note that the same steps can be applied to the python33 collection.) I'm going to work on a RHEL 6 machine throughout this whole tutorial. I'm assuming that readers have basic knowledge of RPM building and Software Collections concept.        

General Notes

This tutorial uses a technique that we call "dependent collections". Dependent collections are a systematic approach to the "how can I extend SCLs?" problem, not a new technology, so don't get scared. We will be extending python27 collection by building another collection that depends on it. This is a general approach that can be applied to most of RHSCL 1.2 collections, although some specific steps may differ from collection to collection (read the details).

Please note, that the "dependent collections" approach as demonstrated here won't work with collections from RHSCL 1.0, so update to the latest version.

Preparing the Environment

At first, we must install some development tools to be able to actually build RPMs and SCLs:

sudo yum install rpmdevtools scl-utils-build

Then we need to install the python27 collection and its "-scldevel" package (make sure you're subscribed to the correct RHSCL channel):

sudo yum install python27 python27-scldevel

The python27-scldevel package contains some macros useful for building collections dependent on python27.

As the last step, we will create ~/rpmbuild directory with all necessary subdirectories, so that we have a place to do the builds:

rpmdev-setuptree

Building a New Collection

Building a new Software Collection consists of two steps:

• creating a metapackage
• converting and building all packages we want to be in the new SCL

Creating a Metapackage

A Software Collection metapackage is a package that defines the Software Collection's name (since we'll be packaging Versiontools 1.9.1, let's go with vt191) and holds RPM macros related to the collection and the "enable" scriptlet. Create

~/rpmbuild/SPECS/vt191.spec

with this content (see comments inline for explanation):

# define name of the scl
%global scl vt191
%scl_package %scl

# Defaults for the values for the python27 Software Collection. These
# will be used when python27-scldevel is not in the buildroot
%{!?scl_python:%global scl_python python27}
%{!?scl_prefix_python:%global scl_prefix_python %{scl_python}-}

# Only for this build, we need to override default __os_install_post, because
# the default one would find /opt/.../lib/python2.7/ and rpmbuild would
# try to bytecompile with the system /usr/bin/python2.7 (which doesn't exist)
%global __os_install_post %{%{scl_python}_os_install_post}
# Similarly, override __python_requires for automatic dependency generator
%global __python_requires %{%{scl_python}_python_requires}

# The directory for site packages for this Software Collection
%global vt191_sitelib %(echo %{python27python_sitelib} | sed 's|%{scl_python}|%{scl}|')

Summary: Package that installs %scl
Name: %scl_name
Version: 1
Release: 1%{?dist}
License: GPLv2+
BuildRequires: scl-utils-build
# Always make sure that there is the python27-sclbuild
# package in the buildroot
BuildRequires: %{scl_prefix_python}scldevel
# Require python27-python-devel, we will need macros from that package
BuildRequires: %{scl_prefix_python}python-devel
Requires: %{scl_prefix}python-versiontools

%description
This is the main package for %scl Software Collection.

%package runtime
Summary: Package that handles %scl Software Collection.
Requires: scl-utils
Requires: %{scl_prefix_python}runtime

%description runtime
Package shipping essential scripts to work with %scl Software Collection.

%package build
Summary: Package shipping basic build configuration
Requires: scl-utils-build
# Require python27-scldevel so that there is always access
# to the %%scl_python and %%scl_prefix_python macros in builds for this Software
# Collection
Requires: %{scl_prefix_python}scldevel

%description build
Package shipping essential configuration macros to build %scl Software Collection.

%prep
%setup -c -T

%install
%scl_install

# Create the enable scriptlet that:
# - Adds an additional load path for the Python interpreter.
# - Runs scl_source so that you can run:
# scl enable vt191 "bash"
# instead of:
# scl enable python27 vt191 "bash"

cat >> %{buildroot}%{_scl_scripts}/enable << EOF
. scl_source enable %{scl_python}
export PYTHONPATH=%{vt191_sitelib}${PYTHONPATH:+:${PYTHONPATH}}
EOF

mkdir -p %{buildroot}%{vt191_sitelib}

# - Enable Software Collection-specific bytecompilation macros from
# the python27-python-devel package.
# - Also override the %%python_sitelib macro to point to the vt191 Software
# Collection.
# - If you plan to add some architecture-dependent packages, you will also need to override
# the %%python_sitearch macro.

cat >> %{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config << EOF
%%scl_package_override() %%{expand:%{?python27_os_install_post:%%global __os_install_post %%python27_os_install_post}
%%global __python_requires %%python27_python_requires
%%global __python_provides %%python27_python_provides
%%global __python %python27__python
%%global __python2 %python27__python
%%global python_sitelib %vt191_sitelib
%%global python2_sitelib %vt191_sitelib
}
EOF

%files

%files runtime
%scl_files

%scl_files
%vt191_sitelib

%files build
%{_root_sysconfdir}/rpm/macros.%{scl}-config

%changelog
* Wed Jan 22 2014 John Doe <jdoe@example.com> - 1-1
- Initial package.

Build this package by running:

cd ~/rpmbuild/SPECS
rpmbuild -ba vt191.spec

And then install vt191-build and vt191-runtime so that we can build packages for vt191 collections:

sudo yum install ~/rpmbuild/RPMS/x86_64/vt191-{build,runtime}-1-1.el6.x86_64.rpm

Converting and Building Packages

We will be building Versiontools 1.9.1 as the only package for our collection. The specfile -

~/rpmbuild/SPECS/python-versiontools.spec

• looks like this (again, see comments inline):

%{?scl:%scl_package python-versiontools}
%{!?scl:%global pkg_name %{name}}

%global pypi_name versiontools

Name: %{?scl_prefix}python-versiontools
Version: 1.9.1
Release: 1%{?dist}
Summary: Smart replacement for plain tuple used in __version__

License: LGPLv3
URL: https://launchpad.net/versiontools
Source0: http://pypi.python.org/packages/source/v/versiontools/versiontools-1.9.1.tar.gz

BuildArch: noarch
BuildRequires: %{?scl_prefix_python}python-devel
BuildRequires: %{?scl_prefix_python}python-setuptools
%{?scl:BuildRequires: %{scl}-build %{scl}-runtime}
%{?scl:Requires: %{scl}-runtime}

%description
Smart replacement for plain tuple used in __version__

%prep
%setup -q -n %{pypi_name}-%{version}

%build
%{?scl:scl enable %{scl} "}
%{__python} setup.py build
%{?scl:"}

%install
# Explicitly specify --install-purelib %{python_sitelib}, which is now overriden
# to point to vt191, otherwise Python will try to install into the python27
# Software Collection site-packages directory
%{?scl:scl enable %{scl} "}
%{__python} setup.py install -O1 --skip-build --root %{buildroot} --install-purelib %{python_sitelib}
%{?scl:"}

%files
%{python_sitelib}/%{pypi_name}*

%changelog
* Wed Jan 22 2014 John Doe <jdoe@example.com> - 1.9.1-1
- Built for vt191 SCL.

If you're creating lots of packages, try using pyp2rpm for creating specfiles from Python packages and spec2scl to make them SCL ready. Also note, that when converted properly, specfiles can be used for both SCL and non-SCL builds.

You'll need to download the actual source file from PyPI:

wget -p --no-directories -P ~/rpmbuild/SOURCES https://pypi.python.org/packages/source/v/versiontools/versiontools-1.9.1.tar.gz

Now you can build Versiontools:

rpmbuild -ba python-versiontools.spec

Then install them:

sudo yum install ~/rpmbuild/RPMS/noarch/vt191-python-versiontools-1.9.1-1.el6.noarch.rpm

And see that everything actually works:

scl enable vt191 "python -c 'import versiontools; print(versiontools.__file__)'"

The above command will print out the full path to the file that Versiontools are loaded from:

/opt/rh/vt191/root/usr/lib/python2.7/site-packages/versiontools/__init__.pyc

Note, that precisely the same approach also works for python33 collection from RHSCL 1.2. This means that you can rebuild vt191 collection as e.g. vt191-p33 without any modifications other than replacing the collection name and interpreter version in macros from vt191.spec.