Testing Farm as GitHub Action: User stories

This article aims to show the possibilities of Testing Farm as GitHub Action and how to configure it. Testing Farm as GitHub Action is a GitHub Action for executing tests on the Testing Farm Service. You can check out previous articles about this action, such as Schedule tests the GitOps way with Testing Farm as GitHub Action, or Test GitHub projects with GitHub Actions and Testing Farm.

The prerequisite for this action is to have TMT plans in GitHub or GitLab. For inspiration, look at our repositories on GitHub or GitLab. These repositories are focused on testing apps and stacks containers here.

Minimal configuration

The minimal configuration that can be used in the upstream project is as follows:

name: Testing farm PR triggered tests
on:
  pull_request_target:
    types: [opened, synchronize, reopened]


jobs:
  tf:
    runs-on: ubuntu-latest
    permissions:
    contents: read
    pull-requests: write
    statuses: write

    steps:
      - name: Get User Permission
       id: checkAccess
       uses: actions-cool/check-user-permission@v2
       with:
          require: write
          username: ${{ github.triggering_actor }}
       env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

   - name: Check User Permission
       if: steps.checkAccess.outputs.require-result == 'false'
       run: |
          echo "${{ github.triggering_actor }} does not have permissions on this repo."
          exit 1

   - name: Check out code
       uses: actions/checkout@v4
       with:
          ref: ${{ github.event.pull_request.head.sha }}

   - name: Schedule test on Testing Farm
       uses: sclorg/testing-farm-as-github-action@v3
       with:
          api_key: ${{ secrets.TF_API_KEY }}
          tmt_plan_regex: smoke
          pull_request_status_name: Smoke test

The configuration for updating statuses in each pull request looks like this:

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - name: Schedule test on Testing Farm
        uses: sclorg/testing-farm-as-github-action@v3
        with:
          api_key: ${{ secrets.TF_API_KEY }}
          tmt_plan_regex: "smoke"
          pull_request_status_name: "Some Description"
          update_pull_request_status: "true"

Once the tests are finished, you will see on the GitHub status window results, as shown in Figure 1.

Create GitHub comment

To enable dynamic comments on pull request showing results, use the following configuration:

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - name: Schedule test on Testing Farm
        uses: sclorg/testing-farm-as-github-action@v3
        with:
          api_key: ${{ secrets.TF_API_KEY }}
          tmt_plan_regex: "smoke"
          pull_request_status_name: "Smoke test"
          create_issue_comment: "true"

Once the tests are finished you will see the pull request GitHub comment, as shown in Figure 2.

Once the tests are executed again, the corresponding GitHub comment is updated.

Create GitHub Job summary

To see more logs, status, and the other information in the GitHub Job summary, use following configuration:

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - name: Schedule test on Testing Farm
        uses: sclorg/testing-farm-as-github-action@v3
        with:
          api_key: ${{ secrets.TF_API_KEY }}
          tmt_plan_regex: "smoke"
          pull_request_status_name: "Smoke test"
          create_github_summary: "true"

An example of GitHub Job summary is shown in Figure 3.

GitHub Actions workflow with matrix testing

To create a matrix job with action, use the following configuration:

jobs:
  tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
   matrix:
       os: ["fedora", "c9s", "c10s"]
       include:
       - os: fedora
           context: Fedora
           compose: Fedora-40
       - os: c9s
           context: CentOS Stream 9
           compose: CentOS-Stream-9
       - os: c10s
           context: CentOS Stream 10
           compose: CentOS-Stream-10
    steps:
      - name: Schedule test on Testing Farm
        uses: sclorg/testing-farm-as-github-action@v3
        with:
          api_key: ${{ secrets.TF_API_KEY }}
          tmt_plan_regex: smoke
          pull_request_status_name: ${{ matrix.context }}
          update_pull_request_status: true
          compose: ${{ matrix.compose }}
          variables: ${{ matrix.os }}

It will spawn a Testing Farm job for each combination in the matrix (Figure 4).

Variable ${{ matrix.os }} is delivered to the Testing Farm host machine, where it is accessible for building, tests, and whatever you would like to do. 

GitHub Action with hardware definition

To create a matrix job with hardware definition, e.g., for AI systems, use the following configuration:

jobs:
 tests:
   runs-on: ubuntu-latest
   strategy:
     fail-fast: false
     matrix:
         os: ["fedora", "c9s", "c10s"]
         include:
         - os: fedora
             context: Fedora
             compose: Fedora-40
             tmt_hardware: '{"gpu": {"device-name": "GK210 (Tesla K80)", "vendor-name": "NVIDIA"}}'
         - os: c9s
             context: CentOS Stream 9
             compose: CentOS-Stream-9
             tmt_hardware: “”
         - os: c10s
             context: CentOS Stream 10
             compose: CentOS-Stream-10
             tmt_hardware: “”
   steps:
     - name: Schedule test on Testing Farm
       uses: sclorg/testing-farm-as-github-action@v3
       with:
         api_key: ${{ secrets.TF_API_KEY }}
         tmt_plan_regex: smoke
         pull_request_status_name: ${{ matrix.context }}
         update_pull_request_status: true
         compose: ${{ matrix.compose }}
         variables: ${{ matrix.os }}
         tmt_hardware: "${{ matrix.tmt_hardware }}"

It will spawn a Testing Farm job with specific hardware. In this case, GPU can be used for AI systems.

Conclusion

Testing Farm as GitHub Action lets you avoid the tedious work of setting up a testing infrastructure, writing a lot of GitHub Action workflows, and handling PR statuses.

You can see that the action is widely configured from minimal configuration, over showing results in GitHub status window, and results in a lone table view overview, to a specific hardware configuration—in this case, showing GPUs mainly used for AI host systems.