Test GitHub projects with GitHub Actions and Testing Farm

Every project on GitHub that's destined for Red Hat Enterprise Linux (RHEL), Fedora, or CentOS should be tested before its changes are synced into a Git distribution repository (dist-git). It's important to catch problems before delivering software to customers, and help quality assurance teams catch errors.

Testing Farm is an open source testing system offered as a service. Testing Farm’s idea is similar to Compile Farms, but with a focus on executing automated tests. Its mission is to provide a reliable and scalable service for executing automated tests from various users, such as Fedora CI, RHEL CI, Packit, and others. The entry point for our users is an HTTP-based API. Testing Farm scales across various infrastructures, including private and public clouds.

The Red Hat Software Collections team has developed a way that you can use our Testing Farm in upstream GitHub repositories. Keep reading to learn how to use Testing Farm to improve your project with internal tests and catch errors before products are delivered to customers.

Note: This article assumes you are familiar with using GitHub Actions.

Get started with Testing Farm and GitHub Actions

Testing Farm has an HTTP-based API that you use to manage testing jobs. Connecting to the Testing Farm API requires two (on demand) artifacts:

• A Testing Farm API key, either for public or private clouds or both
• A URL for Testing Farm requests

To get the API key and request URL, see the Testing Farm onboarding document. Once you've acquired these artifacts, save them to your GitHub repository's secrets so they can be used correctly within GitHub Actions.

The minimal steps for configuring a GitHub Action for running tests in Testing Farm are as follows:

1. Write a Test Management Tool/Flexible Metadata Format (TMT/FMF) testing plan for working with Testing Farm
2. Write a GitHub Action that:
   • Specifies when to run
   • Creates user request for Testing Farm
   • Gets the test results

Writing TMT/FMF plans

Testing Farm requires TMT/FMF plans for executing its tests. You will need to provide information about the provisioned machine and prepare that machine before running the tests.

The TMT plan will be uploaded to the Testing Farm machine during the process of the GitHub Action and then be executed directly from it. The plan will prepare a proper test environment and call the specific command for starting the tests. Here's an example of a TMT plan:

summary: TMT/TFT plan for running tests on CentOS 7
description: |
    Run tests on CentOS-7
discover:
    how: shell
    tests:
    - name: Run tests on CentOS-7
        framework: shell
        test: cd /tmp/$REPO_NAME && <call test suite>
        duration: 3h
prepare:
    how: shell
    script: |
          # TODO install packages needed for tests
          git clone $REPO_URL /tmp/$REPO_NAME
          cd /tmp/$REPO_NAME
          git fetch origin +refs/pull/*:refs/remotes/origin/pr/*
          git checkout origin/pr/$PR_NUMBER/head
          git submodule update --init
execute:
    how: tmt
    

Environment variables used in the TMT plan will be explicitly delivered to the Testing Farm later in the GitHub Action.

You can find more details on how to write TMT plans in the official documentation. The Software Collections team's source file repository contains illustrative examples of TMT/FMF testing plans for CentOS 7, Fedora, and CentOS Stream 9.

Write the GitHub Action

As noted, the TMT plan executes tests in the Testing Farm environment. However, you still need a way to let Testing Farm know specifically when a test should be run. For this, you need to send an HTTP POST request to Testing Farm’s API; the request can be supplied directly from within the GitHub Action.

A GitHub Action used in this way maintains continuity of its execution with the user’s activity in the GitHub repository—a pull request, commit, or branch merging, for instance.

Run tests on explicit user request

This approach triggers tests at Testing Farm by commenting with specific text on a pull request. In this example, the text is the string [test]. Only upstream project owners or members are authorized to trigger the tests:

name: upstream tests at Testing Farm
on:
    issue_comment:
      types:
        -created

jobs:
    build:
      name: A job run on explicit user request
      run-ons: ubuntu-20.04
      if: |
        github.event.issue.pull_request
        && contains(github.event.comment.body, '[test]')
        && contains(fromJson('["OWNER", "MEMBER"]'),
        github.event.comment.author_association)
        

Triggering a test

Testing Farm expects POST HTTP JSON requests as input. To handle this in a GitHub Action, install the curl utility on the machine where the action will execute, along with the jq utility for parsing JSON responses:

- name: Schedule a test on Testing Farm
  id: sched_test
  run: |
      # Update ubuntu-20.04 in order to install curl and jq
      sudo apt update && sudo apt -y install curl jq

The JSON code in the request specifies the environment variables that you want to set in the Testing Farm environment. You also need to provide the API_KEY and path to the TMT plan, as shown in the following listing. The request is then sent to a Testing Farm requests URL:

- name: Schedule a test on Testing Farm
  id: sched_test
  run: |
    cat << EOF > request.json
    {
    "api_key": "${{ secrets.API_KEY }}",
    "test": {"fmf": {
      "url": "<URL_TO_TMT_FMF_plan>",
      "ref": "master",
      "name": "centos7"
    }},
    "environments": [{
      "arch": "x86_64",
      "os": {"compose": "CentOS-7"},
      "variables": {
          "REPO_URL": "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY",
          "REPO_NAME": "$GITHUB_REPOSITORY",
          "PR_NUMBER": "${{ github.event.comment.issue_url }}",
          "TEST_NAME": "test"
      }
    }]
    }
    EOF
    curl ${{ secrets.TF_ENDPOINT }}/requests --data @request.json --header "Content-Type: application/json" --output response.json
    

The job ID will be embedded in the HTTP response from Testing Farm, which you can parse from the JSON and later use to query the results:

# Store REQ_ID into outputs for later on usage
req_id=$(jq -r .id response.json)
echo "REQ_ID=$req_id" >> $GITHUB_ENV

Get the test results

You can periodically request the state of your current test job from Testing Farm. The status request is an HTTP GET pointing to the Testing Farm URL endpoint—specifically, to REQ_ID, which you stored in the previous step:

curl ${{ secrets.TF_ENDPOINT }}/requests/${{ env.REQ_ID }}

When the job is completed, this information is stored in the value of the state key. The final test result can be parsed from the result key value.

Test logs can be gathered within the job ID directory at testing-farm.io. Test results can be displayed as a status directly within a pull request with the GitHub status API.

Run the test suite on Testing Farm

At this point, you know how to set up everything you need to execute a test on Testing Farm infrastructure directly from a GitHub repository.

A full Testing Farm/GitHub Actions scheme example like the one described in this article can be found in the GitHub repository for the Software Collections team. This GitHub Action runs tests on Fedora, CentOS 7, CentOS Stream 9, RHEL 7, and RHEL 8. The action requires that the tested repository has access to the Testing Farm API and request URL, stored as GitHub secrets. The GITHUB_TOKEN secret is necessary to connect to the GitHub API.

Testing Farm machines are situated within the Red Hat infrastructure, in case you are using Testing Farm private clouds. Therefore, all tests running on those machines must be checked for vulnerabilities and potentially dangerous behavior. Consequently, you should be sure to check the code in pull requests carefully from the role of owner or member.

Once you review the pull request, assuming you don't catch any problems or find any incorrect code, you can write a comment into the pull request containing the string [test]. Submitting this string triggers the tests.

Important note: This article has described how to set up the GitHub Action so that only the owner or a member of the GitHub organization can trigger the tests, a constraint we introduced for security reasons.

Conclusion

The combination of GitHub Actions and the Testing Farm environment provides a robust and easy-to-configure testing platform. By using GitHub Actions, you can automate responses to activity in a GitHub repository and straightforwardly display test results directly in the same repository.

Testing Farm has an infrastructure of machines available on request, incorporating RHEL, CentOS, and Fedora distributions with wide configuration options adjustable for any use case. Bringing those features together offers developers and testers a great opportunity to detect defects before they enter the product codebase.