Schedule tests the GitOps way with Testing Farm as GitHub Action

Testing Farm is an open source test system that makes it easier to run unit tests on your projects. A previous article, Test GitHub projects with GitHub Actions and Testing Farm, showed you how to use the framework. This article introduces a new GitHub Action named Testing Farm as GitHub Action, which is available on GitHub Marketplace and integrates your tests into a GitOps environment so that you can ensure that tests are run at key points during development.

Note: GitHub Marketplace is a central site where developers can find GitHub Actions along with other resources. Anyone with a GitHub account can publish an Action on GitHub Marketplace. Publishing the Action in the Marketplace brings additional visibility.

Testing Farm is a service maintained by Red Hat, and is currently available to internal services, Red Hat Hybrid Cloud services, and open source projects related to Red Hat products. Any kind of test described with a Test Management Tool (TMT) plan can be executed. Your testing environment can be Red Hat Enterprise Linux, Fedora, CentOS, or CentOS Stream.

We recommend that you read the previous article before this one to learn how Testing Farm works and how it fits into the development cycle.

About Testing Farm as GitHub Action

Testing Farm provides a common interface to many types of tests and makes it easy to run them in a uniform manner, but it's still the developer's job to schedule the tests. With Testing Farm as GitHub Action, you can decide at what points in your development cycle (creating, checking or merging a pull request (PR), etc.) you want to run tests, and have them run automatically by GitHub.

Note: In order to avoid running malicious code on the Testing Farm infrastructure, it is important to have the tests reviewed by an authorized person.

The Action can be used by developers, maintainers, or anyone else who wants to test a repository located on GitHub. We recommend that, to make sure unauthorized code doesn't run in a testing environment, you allow only members and owners of a repository to use the Action.

Testing Farm as GitHub Action is a composite Action, intended to be used from other GitHub Actions. You can find more details about how composite actions work in the GitHub documentation. Testing Farm as GitHub Action schedules tests so that triggers in a GitHub repository run the tests on the Testing Farm infrastructure and, optionally, display their results.

Action inputs

Testing Farm as GitHub Action is highly configurable, but all the variables have defaults except the following, which must be configured by the user:

• api_key: API key used to get access to Testing Farm
• git_url: URL of a GitHub repository with one or more TMT plans

To obtain your API key, send an email to tft@redhat.com as described in Testing Farm's onboarding documentation.

Here's a minimal example of what it would look like to use Testing Farm as GitHub Action on a repository that you have checked out:

- name: Schedule tests Testing Farm
  uses: sclorg/testing-farm-as-github-action@v1
  with:
      api_key: ${{ secrets.API_KEY }}
      git_url: <URL to a TMT plan>

All other input values are optional. They fall into the following groups:

• TMT metadata: Options for configuring the TMT specification, such as the URL for the Git repository with the TMT plan, or a regular expression that selects plans.
• Testing Farm: Options for configuring Testing Farm itself. You can configure the API key, the URL to Testing Farm's API, and the scope (public or private) of the Testing Farm in use.
• Test environment: Options for configuring the operating system and architecture where the test will be run. The supported Linux distributions are Fedora, CentOS (including CentOS Stream), and Red Hat Enterprise Linux 7 and 8. You can also specify the secrets and environment variables needed during test execution.
• Test artifacts: Additional artifacts to install in the test environment. For more information, see Testing Farm's REST API documentation.
• Miscellaneous: Settings for various options, such as whether the PR should be updated with test results after finishing the job and what should be written in the PR status.

More information about the inputs can be found at the Action's GitHub repository.

Action outputs

Each run of Testing Farm as GitHub Action returns two outputs: a req_id and a results_url. You can combine these values to obtain a URL pointing to a log. Test logs and test results from the Testing Farm are collected here in text form.

Optionally, if the event that triggers Testing Farm as GitHub Action is related to a PR, the user can enable a PR status update. This update places a summary of test results in a graphical form directly in the PR. By default, test results are displayed as a status using the GitHub statuses API. Figure 1 shows sample output.

A Testing Farm as GitHub Action example

The following configuration runs tests when a PR is created on the repository. (You can find the full example YAML file in the software collections GitHub repository.) This configuration follows the recommendation that the person running the tests must be an owner or member of the repository:

name: upstream tests at Testing Farm
on:
  issue_comment:
    types:
      - created

jobs:
    build:
      name: A job run on explicit user request
      runs-on: ubuntu-20.04

 if: |
    github.event.issue.pull_request
    && contains(github.event.comment.body, '[test]')
    && contains(fromJson('["OWNER", "MEMBER"]'),
                github.event.comment.author_association)

The Action checks out the repository and switches to the branch of the PR branch:

steps:

- name: Checkout repo
  uses: actions/checkout@v2
  with:
        ref: "refs/pull/${{ github.event.issue.number }}/head"

Now you can schedule tests on Testing Farm using the GitHub Action:

- name: Schedule tests on external Testing Farm by Testing-Farm-as-github-action
  id: github_action
  uses: sclorg/testing-farm-as-github-action@v1
  with:
      api_key: ${{ secrets.TF_API_KEY }}
      git_url: "https://github.com/sclorg/sclorg-testing-farm"
      variables: "REPO_URL=$GITHUB_SERVER_URL/$GITHUB_REPOSITORY;REPO_NAME=$GITHUB_REPOSITORY;PR_NUMBER=${{ github.event.issue.number }};OS=centos7;TEST_NAME=test"
      compose: "CentOS-7"

Conclusion

Testing Farm as GitHub Action lets you avoid the tedious work of setting up a testing infrastructure, writing a lot of GitHub Action workflows, and handling PR statuses. With just a TMT test plan and an API key, you can get access to a whole testing infrastructure that will meet your needs. Various processor architectures and Linux distributions are available as testing environments.

To get started, all you need is to request an api_key from the Testing Farm team, write a simple GitHub workflow, and use the Action.

Your tests are triggered by an event you specify in the configuration file. Logs and results from test execution are collected, reported, and stored in text form and optionally also transparently displayed in the PR status. Testing Farm as GitHub Action makes it easy to test project changes as soon as possible before your project goes to real customers.

If you want to learn more about how GitOps can transform your workflow, check out Getting Started with ArgoCD and OpenShift GitOps Operator, a hands-on lab from Red Hat Developer.