Deploy self-hosted GitHub Actions runners for Red Hat OpenShift

Note: This article is an update on Red Hat’s work in the GitHub Actions ecosystem. If you haven’t read the December press release or the original openshift.com blog post, be sure to check those out for the full context.

To date, Red Hat has released a series of GitHub Actions and an OpenShift starter workflow to ease integrating Red Hat OpenShift with GitHub’s popular CI/CD platform. Since the initial announcement, we’ve been hard at work responding to community feedback to improve our existing actions as well as adding a few new ones.

In addition to our direct work, we've implemented a way to self-host GitHub Actions runners on a Red Hat OpenShift cluster so you can run workflows on your own cluster instead of GitHub's.

If you don’t have access to an OpenShift cluster, we highly recommend checking out the free Developer Sandbox for Red Hat OpenShift. There is also Red Hat CodeReady Containers for a local option that runs on your workstation. Since GitHub runners connect outwardly to GitHub, you can run them locally or behind a firewall without worrying about networking.

Introducing self-hosted runners for Red Hat OpenShift

We've developed a solution for hosting GitHub Actions runners on OpenShift. This means you can run workflow jobs in containers on OpenShift, which has many advantages, as we'll see in this section.

By default, GitHub provides infrastructure that users can run their Actions workflow jobs on. Many Actions users are satisfied with the hosted solution and do not consider other options.

However, GitHub also provides tools for users to run their own Actions runners. Known as self-hosted runners, these are an alternative to the GitHub runners. Self-hosted runners can be almost any physical or virtual machine, with the runner software supporting many operating systems and architectures.

We are specifically looking to build runners inside of containers that we can then launch on any container runtime—namely, Red Hat OpenShift, and Kubernetes in general.

Why use a self-hosted runner?

Using GitHub's hosted runners is easy and free for public repositories. The GitHub runners are a core component that has lead to Actions' rapid adoption as it becomes the go-to free CI/CD platform for new, open source projects. However, there are several advantages to hosting your own runners.

Benefit #1: Self-hosted runners work with GitHub Enterprise Server

GitHub Enterprise (GHE) Server does not support GitHub-hosted runners. GHE Server runs on-premises, on the client company’s infrastructure.

So, if you're an enterprise that keeps everything in-house, you must use self-hosted runners.

Benefit #2: Self-hosted runners have no usage limits

GitHub-hosted runners are not always free for private repositories. An active project using a private repository is likely to run out of minutes fairly quickly. You don't want to waste developer resources trying to optimize your workflows or disabling them altogether to save money.

Self-hosted runners do not have any resource limits, other than what your infrastructure can provide. Our workflows are also set up to make it easy to create a runner, run a job, and tear down the runner, keeping your resource usage down. Or, if workflow execution speed is the priority, you can leave the runner online.

Benefit #3: Self-hosted runners have persistent disks

GitHub-hosted runners give each workflow run a clean environment. While there are advantages to clearing state between workflows, you sacrifice the ability to persist data between workflows. Actions has good caching functionality, but it has time and space limits, and cannot easily cache container images.

For example, your workflows probably have to download a runtime or SDK, your customized development tool stack, all of your project's dependencies, and a set of container images.

Self-hosted runners can have all the tools and dependencies you need preinstalled, baked into the Containerfile. They can cache container images, and if you're building your own images, they will even cache intermediate image layers. They even cache the images used to run containerized actions, which have to be rebuilt for every workflow job on GitHub runners.

Overall, the workflow runtime saved by a proper persistent disk can be significant.

OpenShift Actions runners overview

We have developed a set of tools that can install GitHub Actions runners onto an existing Red Hat OpenShift or Kubernetes cluster. They include:

• A set of container images that run the GitHub Actions runner and work on Red Hat OpenShift, the OpenShift Actions runners.
• A Helm chart to deploy pods from those images, the OpenShift Actions Runner Chart.
• An Action to automate the Helm install, building the runner management into your workflows, the OpenShift Actions Runner Installer.

We will focus on the installer, which is meant to be the most straightforward way to set up runners.

OpenShift Actions Runner Installer

The easiest way to add self-hosted runners to your Red Hat OpenShift environment is to use the OpenShift Actions Runner Installer. The configuration required for this action is minimal.

1. Create secrets containing your OpenShift cluster API server URL and credentials, and create a workflow step to log into your cluster.
   • See oc-login for instructions on how to authenticate to Red Hat OpenShift from an Actions workflow.
   • In the example that follows, the URL and access token secrets are called OPENSHIFT_SERVER and OPENSHIFT_TOKEN, respectively.
2. Create an Actions Secret containing a GitHub Personal Access Token (PAT) with repo permissions.
   • If you want to install the runners at the organization level, the token must also have admin:org permission.
   • You will need to have administrator permissions on any repository or organization you want to add runners to.
   • In the example, the token secret is called GITHUB_PAT.

The workflow will look like this:

name: Build Quarkus project on OpenShift
on: 
  push: 
  
jobs: 
  install_runner: 
    name: Install Runner 
    runs-on: ubuntu-20.04 

    steps: 
      - name: Log in to OpenShift 
        uses: redhat-actions/oc-login@v1 
        with: 
          openshift_server_url: ${{ secrets.OPENSHIFT_SERVER }} 
          openshift_token: ${{ secrets.OPENSHIFT_TOKEN }} 
      
      - uses: redhat-actions/openshift-actions-runner-installer@v1 
        with: 
          github_pat: ${{ secrets.GITHUB_PAT }}     # Personal access token with organization permissions
          runner_location: redhat-actions                               # Install organization level runners
          runner_image: quay.io/redhat-github-actions/java-runner-11    # Use our prebuilt Java runner image
          runner_labels: openshift, java                                
  
  build-quarkus: 
    name: Self Hosted Quarkus Build and Test
    
    runs-on: [ self-hosted, openshift, java ]             # Use the same labels we gave the runner above    
    needs: install_runner                                 # Wait until the job above completes

    env:
      WORKDIR: getting-started                  

    steps:
      - uses: actions/checkout@v2                         # Checkout a java project to build
        with:
          repository: redhat-actions/quarkus-quickstarts

      # https://github.com/redhat-actions/quarkus-quickstarts/tree/master/getting-started#readme

      - name: Install and build
        run: ./mvnw install -ntp
        working-directory: ${{ env.WORKDIR }}

      - name: Run tests
        run: ./mvnw test
        working-directory: ${{ env.WORKDIR }}

      - name: Upload jar artifacts
        uses: actions/upload-artifact@v2
        with:
          name: app-jar-files.zip
          path: ${{ env.WORKDIR }}/target/*.jar

That's all it takes to set up the runner. The runners appear under the organization or repository's settings, under the Actions menu, in the Runners subsection.

There are plenty of ways you can configure the runner-installer to use your own images, add descriptive labels to your runner, or scale up the number of replicas. See the action's list of inputs and the Helm chart README.

If you'd prefer to keep the runner management separate from the project's workflows, you can use the repository dispatch event to trigger another workflow once the runner is ready. The workflow can be in the same repository or a different one. Be sure to check out the examples.

By default, the runner won't be recreated every workflow run. If a runner matching the same image, tag, and labels is already present, the installer will exit. Or, if you prefer to keep a clean state and minimize resource usage, you can remove the runner at the end of each run.

If you prefer to manage your runner operations yourself, you can use the Helm chart directly.

OpenShift Actions runner images

The prebuilt runner images are relatively simple to keep image size and complexity manageable. Since containers run without root privileges without modifications to the Helm chart, any setup steps that require root have to be run at the image build time. This means images have to be built with a specific toolchain in mind.

As of this time, we’ve built the following images. The Containerfiles and documentation can be found in the runners repository:

• The base image, which is a Fedora image, some core utilities, and the actions runner application.
  • All our other images are built from this one, and it is the one you should use as a base image when building your own image.
• The Buildah and Podman image, which runs these tools rootlessly. This image can run the OpenShift starter workflow.
  • This pod must run with higher privileges than the OpenShift default.
  • Instructions for setting up a Service Account that can create these runners with the necessary permissions can be found in the Buildah/Podman image README.
• A Kubernetes (K8s) tools image, which bundles a set of common OpenShift and Kubernetes-related command-line interfaces (CLIs).
  • Since the runner is a pod, it is already authenticated to the OpenShift cluster, but with no permissions by default.
  • Similar to the Buildah image, you can override the Service Account used to deploy the pod to be able to read and edit resources in the cluster.
• A Node.js 14 image.
  • This image is useful for running CI/CD workflows for Node.js projects.
  • It serves as an example of adding a runtime and development tools to the base runner.
• A Java Development Kit 11 image.
  • This image is helpful for running CI/CD workflows for Java projects and is used for the Quarkus workflow in the example in the previous section.
  • It also serves as an example of adding a runtime and development tools to the base runner.

You can use these images or refer to them as examples of how to extend the base image and create your own. Instructions are available in the base runner README.

We are happy to help you create an image that can run your workflow. In addition, if you believe others may want to use your image, open an issue, and we can explore merging it into our organization.

Conclusion

Give the new actions and the OpenShift self-hosted runner a try. We strive to make the integration between Red Hat OpenShift and GitHub Actions as easy and useful as possible, and hosting workflow runners in your OpenShift cluster is an excellent next step.

We are always looking for community feedback. If you have a bug report, enhancement request, question, action idea, or new use case, open an issue on any of our repositories, and we will be sure to take a look.