Our infrastructure and how we continuously integrate and deliver (CI/CD) our code changes with each piece of work we finish. We fully expect that it will change even more as we bring up A|B functionality & enhance our testing process. However, we thought it would be fun to give a closer look into the team's pipeline from checking out code to integration testing.
Check out code
We use a combination of github.com and an internal installation of gitolite, depending on what we are storing. Obviously, external code is stored on github but we also store the code in gitolite with internal configurations.
The developer checks out code from the appropriate location, writes unit tests, writes code to pass unit tests, and checks everything back in. Upon check-in to github, our external build processes are kicked off.
Build (external) + unit test
Travis CI is our current tool of choice for external continuous integration. The build includes coverage.py, which evaluates the % of code coverage for each release engine component. If it falls below an overall 80%, the build will be considered a failure.
Travis build status is displayed on a team radiator in real time. The radiator cycles through multiple information screens using a Red Hat IT developed tool called RNOC. Our information radiator includes not only build information for all of the tools we support, but also outstanding defect lists, current iteration status, our overall sprint goals, and team improvement goals.
Build (internal)
After Travis has had a successful build and we are ready to deploy our code the developer logs into Jenkins and clicks the 'build now' button. This process kicks off the internal building, packaging, and pushing of artifacts.
Package & Push RPMs
The Jenkins job first uses Mock to package RPMs and then Juicer to upload/promote the RPMs where they are needed to be stored in Pulp.
- Mock is a tool for chroot based package building.
- Juicer is a pulp administration tool that helps us create juicer carts (collections of related RPMs; for example, all of the RPMs for a deployment), upload, and promote as needed.
- Pulp is used to store the RPMs in yum repositories.
Build Image
Once the RPM is created, the same Jenkins job will build a docker image, using RHEL 6.5 containers on RHEL7, that will contain the newest RPM along with the unique configurations for the different environments.
Push Image
The docker image, which now contains the newest RPM and environmental configuration, will then be uploaded into an internal docker repository. The repo allows the team to either perform local testing on their laptops or deploy to the docker host. Once the RPMs & images work is complete, Jenkins will notify the team via email.
Images are deployed throughout the different environments manually which ensures proper testing will take place in lower environments (also a jenkins job) before anything goes to stage and production.
With the exception of the two manual steps to kick off our internal build process and pushing the docker images out to the different environments, the process takes around 10 minutes.
Integration Testing
At this time, our integration testing process is simple. We continuously run a sample playbook in our development environment every 30 minutes and will get notifications in IRC upon start, and completion or failure of the playbook run.
Future State
- We were wanting to have our internal code built and images deployed automatically to our development environment, but were waiting to build our stage and production environments before we enhance that process. Happily, we just finished stage & production with the end of the last sprint.
- Eventually we will integrate our release engine tooling into this process -- we first want an A|B environment + feature flagging running -- with the intention of letting the release engine orchestrate its own release process. So meta! So Inception!
We know from experience in Red Hat IT that everyone does CI/CD a little bit differently. We'd love to hear what you are doing and how you are doing it. Additionally, over the next few blogs, the team will be talking about other things we are working on, including an enhancement we made to our unit tests which makes them compatible with Software Collections.
Read other articles from Team Inception.
