Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Test GitHub projects with GitHub Actions and Testing Farm

March 9, 2022
Petr Hracek Zuzana Miklankova
Related topics:
Automation and managementDevOpsGitOpsLinux
Related products:
Red Hat Enterprise Linux

Share:

    Every project on GitHub that's destined for Red Hat Enterprise Linux (RHEL), Fedora, or CentOS should be tested before its changes are synced into a Git distribution repository (dist-git). It's important to catch problems before delivering software to customers, and help quality assurance teams catch errors.

    Testing Farm is an open source testing system offered as a service. Testing Farm’s idea is similar to Compile Farms, but with a focus on executing automated tests. Its mission is to provide a reliable and scalable service for executing automated tests from various users, such as Fedora CI, RHEL CI, Packit, and others. The entry point for our users is an HTTP-based API. Testing Farm scales across various infrastructures, including private and public clouds.

    The Red Hat Software Collections team has developed a way that you can use our Testing Farm in upstream GitHub repositories. Keep reading to learn how to use Testing Farm to improve your project with internal tests and catch errors before products are delivered to customers.

    Note: This article assumes you are familiar with using GitHub Actions.

    Get started with Testing Farm and GitHub Actions

    Testing Farm has an HTTP-based API that you use to manage testing jobs. Connecting to the Testing Farm API requires two (on demand) artifacts:

    • A Testing Farm API key, either for public or private clouds or both
    • A URL for Testing Farm requests

    To get the API key and request URL, see the Testing Farm onboarding document. Once you've acquired these artifacts, save them to your GitHub repository's secrets so they can be used correctly within GitHub Actions.

    The minimal steps for configuring a GitHub Action for running tests in Testing Farm are as follows:

    1. Write a Test Management Tool/Flexible Metadata Format (TMT/FMF) testing plan for working with Testing Farm
    2. Write a GitHub Action that:
      • Specifies when to run
      • Creates user request for Testing Farm
      • Gets the test results

    Writing TMT/FMF plans

    Testing Farm requires TMT/FMF plans for executing its tests. You will need to provide information about the provisioned machine and prepare that machine before running the tests.

    The TMT plan will be uploaded to the Testing Farm machine during the process of the GitHub Action and then be executed directly from it. The plan will prepare a proper test environment and call the specific command for starting the tests. Here's an example of a TMT plan:

    
    summary: TMT/TFT plan for running tests on CentOS 7
    description: |
        Run tests on CentOS-7
    discover:
        how: shell
        tests:
        - name: Run tests on CentOS-7
            framework: shell
            test: cd /tmp/$REPO_NAME && <call test suite>
            duration: 3h
    prepare:
        how: shell
        script: |
              # TODO install packages needed for tests
              git clone $REPO_URL /tmp/$REPO_NAME
              cd /tmp/$REPO_NAME
              git fetch origin +refs/pull/*:refs/remotes/origin/pr/*
              git checkout origin/pr/$PR_NUMBER/head
              git submodule update --init
    execute:
        how: tmt
        

    Environment variables used in the TMT plan will be explicitly delivered to the Testing Farm later in the GitHub Action.

    You can find more details on how to write TMT plans in the official documentation. The Software Collections team's source file repository contains illustrative examples of TMT/FMF testing plans for CentOS 7, Fedora, and CentOS Stream 9.

    Write the GitHub Action

    As noted, the TMT plan executes tests in the Testing Farm environment. However, you still need a way to let Testing Farm know specifically when a test should be run. For this, you need to send an HTTP POST request to Testing Farm’s API; the request can be supplied directly from within the GitHub Action.

    A GitHub Action used in this way maintains continuity of its execution with the user’s activity in the GitHub repository—a pull request, commit, or branch merging, for instance.

    Run tests on explicit user request

    This approach triggers tests at Testing Farm by commenting with specific text on a pull request. In this example, the text is the string [test]. Only upstream project owners or members are authorized to trigger the tests:

    
    name: upstream tests at Testing Farm
    on:
        issue_comment:
          types:
            -created
    
    jobs:
        build:
          name: A job run on explicit user request
          run-ons: ubuntu-20.04
          if: |
            github.event.issue.pull_request
            && contains(github.event.comment.body, '[test]')
            && contains(fromJson('["OWNER", "MEMBER"]'),
            github.event.comment.author_association)
            

    Triggering a test

    Testing Farm expects POST HTTP JSON requests as input. To handle this in a GitHub Action, install the curl utility on the machine where the action will execute, along with the jq utility for parsing JSON responses:

    
    - name: Schedule a test on Testing Farm
      id: sched_test
      run: |
          # Update ubuntu-20.04 in order to install curl and jq
          sudo apt update && sudo apt -y install curl jq
    
    

    The JSON code in the request specifies the environment variables that you want to set in the Testing Farm environment. You also need to provide the API_KEY and path to the TMT plan, as shown in the following listing. The request is then sent to a Testing Farm requests URL:

    
    - name: Schedule a test on Testing Farm
      id: sched_test
      run: |
        cat << EOF > request.json
        {
        "api_key": "${{ secrets.API_KEY }}",
        "test": {"fmf": {
          "url": "<URL_TO_TMT_FMF_plan>",
          "ref": "master",
          "name": "centos7"
        }},
        "environments": [{
          "arch": "x86_64",
          "os": {"compose": "CentOS-7"},
          "variables": {
              "REPO_URL": "$GITHUB_SERVER_URL/$GITHUB_REPOSITORY",
              "REPO_NAME": "$GITHUB_REPOSITORY",
              "PR_NUMBER": "${{ github.event.comment.issue_url }}",
              "TEST_NAME": "test"
          }
        }]
        }
        EOF
        curl ${{ secrets.TF_ENDPOINT }}/requests --data @request.json --header "Content-Type: application/json" --output response.json
        

    The job ID will be embedded in the HTTP response from Testing Farm, which you can parse from the JSON and later use to query the results:

    
    # Store REQ_ID into outputs for later on usage
    req_id=$(jq -r .id response.json)
    echo "REQ_ID=$req_id" >> $GITHUB_ENV
    
    

    Get the test results

    You can periodically request the state of your current test job from Testing Farm. The status request is an HTTP GET pointing to the Testing Farm URL endpoint—specifically, to REQ_ID, which you stored in the previous step:

    
    curl ${{ secrets.TF_ENDPOINT }}/requests/${{ env.REQ_ID }}
    

    When the job is completed, this information is stored in the value of the state key. The final test result can be parsed from the result key value.

    Test logs can be gathered within the job ID directory at testing-farm.io. Test results can be displayed as a status directly within a pull request with the GitHub status API.

    Run the test suite on Testing Farm

    At this point, you know how to set up everything you need to execute a test on Testing Farm infrastructure directly from a GitHub repository.

    A full Testing Farm/GitHub Actions scheme example like the one described in this article can be found in the GitHub repository for the Software Collections team. This GitHub Action runs tests on Fedora, CentOS 7, CentOS Stream 9, RHEL 7, and RHEL 8. The action requires that the tested repository has access to the Testing Farm API and request URL, stored as GitHub secrets. The GITHUB_TOKEN secret is necessary to connect to the GitHub API.

    Testing Farm machines are situated within the Red Hat infrastructure, in case you are using Testing Farm private clouds. Therefore, all tests running on those machines must be checked for vulnerabilities and potentially dangerous behavior. Consequently, you should be sure to check the code in pull requests carefully from the role of owner or member.

    Once you review the pull request, assuming you don't catch any problems or find any incorrect code, you can write a comment into the pull request containing the string [test]. Submitting this string triggers the tests.

    Important note: This article has described how to set up the GitHub Action so that only the owner or a member of the GitHub organization can trigger the tests, a constraint we introduced for security reasons.

    Conclusion

    The combination of GitHub Actions and the Testing Farm environment provides a robust and easy-to-configure testing platform. By using GitHub Actions, you can automate responses to activity in a GitHub repository and straightforwardly display test results directly in the same repository.

    Testing Farm has an infrastructure of machines available on request, incorporating RHEL, CentOS, and Fedora distributions with wide configuration options adjustable for any use case. Bringing those features together offers developers and testers a great opportunity to detect defects before they enter the product codebase.

    Last updated: March 18, 2024

    Related Posts

    • Deploy self-hosted GitHub Actions runners for Red Hat OpenShift

    • Automate dependency analytics with GitHub Actions

    • Managing GitOps control planes for secure GitOps practices

    • A developer's guide to CI/CD and GitOps with Jenkins Pipelines

    Recent Posts

    • Assessing AI for OpenShift operations: Advanced configurations

    • OpenShift Lightspeed: Assessing AI for OpenShift operations

    • OpenShift Data Foundation and HashiCorp Vault securing data

    • Axolotl meets LLM Compressor: Fast, sparse, open

    • What’s new for developers in Red Hat OpenShift 4.19

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue