This article aims to show the possibilities of Testing Farm as GitHub Action and how to configure it. Testing Farm as GitHub Action is a GitHub Action for executing tests on the Testing Farm Service. You can check out previous articles about this action, such as Schedule tests the GitOps way with Testing Farm as GitHub Action, or Test GitHub projects with GitHub Actions and Testing Farm.
The prerequisite for this action is to have TMT plans in GitHub or GitLab. For inspiration, look at our repositories on GitHub or GitLab. These repositories are focused on testing apps and stacks containers here.
Minimal configuration
The minimal configuration that can be used in the upstream project is as follows:
name: Testing farm PR triggered tests
on:
pull_request_target:
types: [opened, synchronize, reopened]
jobs:
tf:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
statuses: write
steps:
- name: Get User Permission
id: checkAccess
uses: actions-cool/check-user-permission@v2
with:
require: write
username: ${{ github.triggering_actor }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Check User Permission
if: steps.checkAccess.outputs.require-result == 'false'
run: |
echo "${{ github.triggering_actor }} does not have permissions on this repo."
exit 1
- name: Check out code
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Schedule test on Testing Farm
uses: sclorg/testing-farm-as-github-action@v3
with:
api_key: ${{ secrets.TF_API_KEY }}
tmt_plan_regex: smoke
pull_request_status_name: Smoke test
This configuration executes tests in the Testing Farm environment. But you do not have any responses in a pull request.
GitHub status configuration
The configuration for updating statuses in each pull request looks like this:
jobs:
tests:
runs-on: ubuntu-latest
steps:
- name: Schedule test on Testing Farm
uses: sclorg/testing-farm-as-github-action@v3
with:
api_key: ${{ secrets.TF_API_KEY }}
tmt_plan_regex: "smoke"
pull_request_status_name: "Some Description"
update_pull_request_status: "true"
Once the tests are finished, you will see on the GitHub status window results, as shown in Figure 1.
Create GitHub comment
To enable dynamic comments on pull request showing results, use the following configuration:
jobs:
tests:
runs-on: ubuntu-latest
steps:
- name: Schedule test on Testing Farm
uses: sclorg/testing-farm-as-github-action@v3
with:
api_key: ${{ secrets.TF_API_KEY }}
tmt_plan_regex: "smoke"
pull_request_status_name: "Smoke test"
create_issue_comment: "true"
Once the tests are finished you will see the pull request GitHub comment, as shown in Figure 2.
Once the tests are executed again, the corresponding GitHub comment is updated.
Create GitHub Job summary
To see more logs, status, and the other information in the GitHub Job summary, use following configuration:
jobs:
tests:
runs-on: ubuntu-latest
steps:
- name: Schedule test on Testing Farm
uses: sclorg/testing-farm-as-github-action@v3
with:
api_key: ${{ secrets.TF_API_KEY }}
tmt_plan_regex: "smoke"
pull_request_status_name: "Smoke test"
create_github_summary: "true"
An example of GitHub Job summary is shown in Figure 3.
GitHub Actions workflow with matrix testing
To create a matrix job with action, use the following configuration:
jobs:
tests:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
os: ["fedora", "c9s", "c10s"]
include:
- os: fedora
context: Fedora
compose: Fedora-40
- os: c9s
context: CentOS Stream 9
compose: CentOS-Stream-9
- os: c10s
context: CentOS Stream 10
compose: CentOS-Stream-10
steps:
- name: Schedule test on Testing Farm
uses: sclorg/testing-farm-as-github-action@v3
with:
api_key: ${{ secrets.TF_API_KEY }}
tmt_plan_regex: smoke
pull_request_status_name: ${{ matrix.context }}
update_pull_request_status: true
compose: ${{ matrix.compose }}
variables: ${{ matrix.os }}
It will spawn a Testing Farm job for each combination in the matrix (Figure 4).
Variable ${{ matrix.os }} is delivered to the Testing Farm host machine, where it is accessible for building, tests, and whatever you would like to do.
GitHub Action with hardware definition
To create a matrix job with hardware definition, e.g., for AI systems, use the following configuration:
jobs:
tests:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
os: ["fedora", "c9s", "c10s"]
include:
- os: fedora
context: Fedora
compose: Fedora-40
tmt_hardware: '{"gpu": {"device-name": "GK210 (Tesla K80)", "vendor-name": "NVIDIA"}}'
- os: c9s
context: CentOS Stream 9
compose: CentOS-Stream-9
tmt_hardware: “”
- os: c10s
context: CentOS Stream 10
compose: CentOS-Stream-10
tmt_hardware: “”
steps:
- name: Schedule test on Testing Farm
uses: sclorg/testing-farm-as-github-action@v3
with:
api_key: ${{ secrets.TF_API_KEY }}
tmt_plan_regex: smoke
pull_request_status_name: ${{ matrix.context }}
update_pull_request_status: true
compose: ${{ matrix.compose }}
variables: ${{ matrix.os }}
tmt_hardware: "${{ matrix.tmt_hardware }}"
It will spawn a Testing Farm job with specific hardware. In this case, GPU can be used for AI systems.
Conclusion
Testing Farm as GitHub Action lets you avoid the tedious work of setting up a testing infrastructure, writing a lot of GitHub Action workflows, and handling PR statuses.
You can see that the action is widely configured from minimal configuration, over showing results in GitHub status window, and results in a lone table view overview, to a specific hardware configuration—in this case, showing GPUs mainly used for AI host systems.
