Continuous Integration Strategies (Part 2 of 3)

This is part 2 of a three part article on Continuous Integration Strategies:  Simplifying application development on Red Hat® Enterprise Linux®

KEY GOALS

Within a Continuous Integration environment there are some key features that are required to encourage an effective process.

SOURCE CODE REPOSITORY—JUST THE ONE

For the effective operation of CI, it’s essential that all project source code and build components are maintained in a single repository. The repository must contain not just the source code, but all the assets needed to build, deploy, and test the software; this may include test scripts, property/configu- ration files, database schema, install scripts, and third-party libraries.

AN AUTOMATED BUILD

A key feature of CI is that software should be built regularly—possibly as regularly as each check-in of amended source code, but no less frequently than daily. It’s therefore essential that the build be automatic, complete, and quick.

Build automation should be advanced enough to eliminate human intervention entirely in the process. To achieve this the build needs to be complete, so that the end result is an artifact that could be installed into a virgin environment and run. Quick should mean an ultimate target of around 10 minutes—which may require significant efficiency gains to be achieved if, for example, nightly or weekly builds are the current practice. Various approaches can be adopted in migrating to the target build time; for example, the code may offer the option to modularize the build so that only a partial build may be needed for most commits. Then full builds may be run less frequently, maybe nightly or weekly depending on how long they take to execute.

TEST EARLY, TEST THOROUGHLY, REPORT QUICKLY

Testing should start early within the build process itself (sometimes known as a self-testing build), so that failed tests also cause the build to fail. Build testing should occur initially at a unit level, and once the software has been successfully built, end-to-end testing should also be completed. In both cases testing needs to be automated. Once testing is completed, results should be reported quickly so that any defects found can be quickly analyzed and removed.

Adoption of Test Driven Development is strongly encouraged to achieve all the benefits of CI. In this technique the development process starts by creating a test for the desired outcome. Software is then created and executed against that test until the desired outcome is achieved. This approach encourages early testing, but also means the availability of a set of tests that can form the basis of build testing with the CI process.

CI FOR YOUR RED HAT ENTERPRISE LINUX APPLICATIONS

When determining how to setup and configure a CI environment for your Red Hat Enterprise Linux applications you should consider:

• The nature of change within your software and the OS.
• The timetables for updates and improvements to the OS.

CHANGE IN YOUR APPLICATION AND THE OS

Each major release of Red Hat Enterprise Linux is delivered through four life cycle phases— Production 1, 2, and 3, and an Extended Life Phase. In the Production 1 through 3 phases, Red Hat delivers minor releases, for example Red Hat Enterprise Linux 6.0 to 6.x. However, while all pro- duction phases deliver Asynchronous Security Errata (RHSA) and Asynchronous Bug Fix Errata (RHBA), only the Production 1 phase delivers Software Enhancements. From a software developer’s perspective, a version of Red Hat Enterprise Linux in the Production 1 Phase is changing, but one in Production 2 or later is less subject to significant change. For more details, see Red Hat Enterprise Linux Life Cycle.

With this in mind, devising a CI environment for your application means considering the following scenarios:

• Stable application, stable platform — Your application will be changing for security, defect, and similar updates only. The target versions of Red Hat Enterprise Linux are in Production 2 or later and also changing for security and similar minor updates only.
• Stable application, changing platform — Your application is subject to minimal change for security, defect, and similar updates only. However, it is targeted at versions of the OS that are in Production 1 and receiving new functional- ity through software enhancements in addition to maintenance patches. You may also be looking to support the next minor release of the OS.
• Changing application, stable platform — Your application is under active development but targeted to run against versions of Red Hat Enterprise Linux in Production 2 or later and changing for security and similar updates only.
• Changing application, changing platform — Your application is under active development and is targeted at versions of the OS that are in Production 1 and receiving new functionality through software enhancements in addition to main- tenance patches. You may also be looking forward to support for the next point release of the OS.

A key consideration for the CI environment is that the triggers for testing shouldn’t be limited to changes in your applications. Testing should also be triggered when the underlying Red Hat Enterprise Linux environments change, both for asynchronous errata and for minor releases.

TARGETING RED HAT ENTERPRISE LINUX VERSIONS

Red Hat Enterprise Linux contains over 2,000 technology components. A subset of these are covered by API and ABI guarantees. From a developer’s perspective, there are three levels of guarantee:

• Assurance Level 1, where an API/ABI will be guaranteed as forward compatible for three major releases of Red Hat Enterprise Linux.
• Assurance Level 2, where an API/ABI is guaranteed as forward compatible within a major release of Red Hat Enterprise Linux.
• Assurance Level 3 and 4, where, from an application developer’s perspective, no guarantee is provided.

(For more details on these levels, see the Red Hat Enterprise Linux Application Compatibility Specification.)

Note that not all behavior or symbols in a library are covered by API/ABI compatibility — only the defined public interfaces of the library are covered under the API and ABI guarantee: bug-for-bug compatibility between library versions is not guaranteed. Relying on non-public symbols or implementation-specific behavior that is not part of a library’s defined interface may lead to unexpected results.

However, by following a number of recommended practices, you should be able to preserve binary compatibility in your application. The recommended practices include building applications using the public interfaces of a library, packaging applications using the RPM mechanism, and avoiding application design that relies on configuration files provided by Red Hat Enterprise Linux. You can find detailed advice in the “Best practices for preserving binary compatibility” section of the Red Hat  Enterprise Linux Application Compatibility Specification.

Regardless of the approach taken, creating tests for execution in your CI environment that provide coverage of the behavior or symbols used in your application can eliminate concerns about whether or not the software is covered by ABI/API guarantees. And your CI environment will enable you to test the application against each minor release of Red Hat Enterprise Linux from the original target version through to the beta of the next minor release.

During migration to the optimal CI environment, the following techniques may assist where a com- plete set of release tests cannot be achieved.

API/ABI GUARANTEES

Where an application is dependent on APIs/ABIs that have Assurance levels 1 or 2, it might seem that testing needs to be performed only on the baseline target version of the OS. However, it’s recommended that testing is more extensive, with the goal of confirming that the application functions cor- rectly on:

• The original target OS release; for example, this might be Red Hat Enterprise Linux 6.1.
• The latest minor release of the OS, for example Red Hat Enterprise Linux 6.4.
• The next planned minor release, for example Red Hat Enterprise Linux 6.5. (And don’t forget that you can get a beta version of the next minor software release through your Red Hat subscription from the Red Hat Customer Portal.)

This ‘bookend’ approach would, in the context of the API/ABI guarantees, confirm that the application should operate on its original target release through to the next minor release. However, care should be taken to review the API/ABI guarantees carefully to confirm whether testing at intermediate point releases is also advisable.

When taking this approach, if the application fails to function correctly from the same compilation, the issue should be raised as a defect with Red Hat.

You may wish to extend testing beyond the next point release to the next major version of Red Hat Enterprise Linux and/or the most recent Fedora release or the bleeding edge release, Fedora Rawhide—but this is not generally necessary for commercial software releases.

NO API/ABI GUARANTEES

Where an application relies on APIs/ABIs that are not subject to a guarantee (Assurance levels 3 and 4), testing will need to be conducted on all software versions from the implementation baseline through to the latest release, including all errata.

For example, if the application was originally developed for Red Hat Enterprise Linux 6.2, testing will need to be performed on Red Hat Enterprise Linux 6.3, 6.4, and future releases as they become available.

An additional strategy worth considering, when working with APIs/ABIs that aren’t subject to a guarantee, is to package your own versions of these libraries with your application using a software collection.

We hope you enjoyed Part 2 of this article.  Part 3 can be viewed here.