Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Wiping the Slate Clean with the OpenShift Container Platform

 

November 16, 2017
Benjamin Holmes
Related topics:
ContainersDevOpsKubernetes
Related products:
Red Hat OpenShiftRed Hat OpenShift Container Platform

Share:

    With traditional virtualized infrastructure or Infrastructure-as-a-Service, it is common practice to regularly refresh instances back to a known good state. This provides confidence that the application workloads have the correct runtime configuration, no deltas are being introduced, and they can be relied upon to provide value for the business. In these cases, you might use tools such as Ansible or Jenkins, but when we move our application workloads to containers running on OpenShift Container Platform, we can use native tools provided by that platform to achieve the same result.

    But Why Would I Do This?

    Application state can still be written to ephemeral storage within a container - nothing prevents this. This anti-pattern can introduce deltas, not only between initial container states but also between multiple replicas of a Pod. Storing state in the container creates a Snowflake, which negates the benefits of immutability and scalability that they provide. Therefore, it is equally critical to have tools that draw attention to non-compliance with the Image Immutability Principle as it is to adhere to it in the first place.

    In addition, having such a mechanism also brings some added benefits in terms of application architecture. It can serve to highlight whether or not:

    • A container in a Pod is able to gracefully handle enforced shutdown/restart.
    • Other applications and services that rely on this Pod are able to handle its absence gracefully.
    • Enough replicas of this Pod are maintained to provide service continuity during redeployment.

    That All Sounds Reasonable. How Do I Do It?

    Helpfully, OpenShift has the concept of Jobs. Unlike a Replication Controller, a Job will run a Pod with any number of replicas through to completion. An extension of this concept is the CronJob, which is the same as a normal Job, but with the added ability to set an execution schedule. This can be done using the well-established Cron notation. It is a very powerful alternative mechanism for executing short-duration workloads or workloads that may sit idle for long periods.

    With these properties in mind, a CronJob is a perfect mechanism for executing the process that wipes our slate clean. The process needs to be executed on a schedule, it'll live for a few seconds at most, and has a very specific task to carry out. But what does that actually look like in practical terms?

    Several things need to happen as part of the process:

    • Check for valid credentials
    • Authenticate with the OpenShift API using those credentials
    • Redeploy Pods

    All of these tasks can be achieved within a CronJob. That doesn't sound so hard, does it?

    Implementing the Clean Slate Job

    Authentication

    Several assumptions can be made which make the task of implementing this process relatively simple. Certificates, environment metadata, and details on the executing Service Account get injected into a Pod at runtime, allowing a process to be introspective and gain information about its environment. This information is found at location /run/secrets/kubernetes.io/serviceaccount.

    [jboss@kitchensink-3-8fs58 serviceaccount]$
    total 0
    lrwxrwxrwx. 1 root root 13 Nov 10 10:56 ca.crt -> ..data/ca.crt
    lrwxrwxrwx. 1 root root 16 Nov 10 10:56 namespace -> ..data/namespace
    lrwxrwxrwx. 1 root root 21 Nov 10 10:56 service-ca.crt -> ..data/service-ca.crt
    lrwxrwxrwx. 1 root root 12 Nov 10 10:56 token -> ..data/token
    

    The token and the ca.crt files are the most useful here. The token contains the authentication token for the Service Account being used to run the Pod, and the certificate can be used to secure communications with the OpenShift API. This information can be used within a simple script to authenticate the process with the OpenShift API:

    #!/bin/bash
    
    if [ -f /run/secrets/kubernetes.io/serviceaccount/token ]; then
    	TOKEN=`cat /run/secrets/kubernetes.io/serviceaccount/token`
    	PROJECT=`cat /run/secrets/kubernetes.io/serviceaccount/namespace`
    	CA=/run/secrets/kubernetes.io/serviceaccount/ca.crt
    else
    	echo "No token found. Are you running on OpenShift?"
    fi
    
    # Make sure we're logged in
    if [ -n "$TOKEN" ]; then
    	echo "Authenticating with token"
    	oc login $KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT --token=$TOKEN --certificate-authority=$CA
    fi
    

    In addition to the files injected at runtime, the script also makes use of environment variables provided to the Pod by the platform as standard - KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT. These allow the script to identify the OpenShift API, and perform actions against it, without having to either hardcode IP addresses or DNS names into the container or otherwise perform a service lookup.

    Pod Redeployment

    Once authenticated, actions can then be carried out using the OpenShift API. Within the script we are executing as part of the CronJob, a pre-populated list of DeploymentConfigs can be iterated over, and a new deployment initiated:

    # Iterate through DCs, and execute a rollout on them
    if [ -n "$DEPLOYMENT_CONFIGS" ]; then
    	for dc in $(echo $DEPLOYMENT_CONFIGS | sed "s/,/ /g")
    	do
    		echo "--"
    		echo "Wiping the slate of $dc"
    		oc rollout latest dc/$dc -n $PROJECT
    		echo "Done"
    	done
    else
    	echo "No DeploymentConfigs specified. Skipping execution."
    fi

    The list of DeploymentConfigs is provided the DEPLOYMENT_CONFIGS environment variable, which turn is provided through an OpenShift ConfigMap. As nothing about the DeploymentConfig has changed, this initiates a graceful redeployment of the existing container, with the same configuration as the previous DeploymentConfig.

    Job Container

    Now that the content of the Job has been defined, it needs to be packaged up in a format that allows OpenShift to run it. This inevitably means the creation of a container. In order to keep this container as lightweight as possible, the RHEL Atomic Base Image can be used. The simplest way to build this container is to provide a Dockerfile that can be built and run on the OpenShift platform:

    FROM registry.access.redhat.com/rhel7-atomic
    ...
    ADD scripts/clean-slate.sh /opt/app/scripts/
    
    RUN microdnf --enablerepo=rhel-7-server-rpms \
    			install tar gzip --nodocs ;\
    			microdnf clean all ;\
    			curl --retry 5 -Lso /tmp/client-tools.tar.gz https://github.com/openshift/origin/releases/download/v3.6.0/openshift-origin-client-tools-v3.6.0-c4dd4cf-linux-64bit.tar.gz ;\
    			tar zxf /tmp/client-tools.tar.gz --strip-components=1 -C /usr/local/bin
    ...		
    CMD ["/opt/app/scripts/clean-slate.sh"]

    NOTE - Sections omitted for brevity.

    This Dockerfile does a number of things:

    1. Add the script created above into the image.
    2. Uses Microdnf to satisfy the dependencies required to install the OpenShift Command Line Tools.
    3. Install the OpenShift Command Line Tools.
    4. Set the CMD point (the process to be run in the container) to be the script we've added to the image.

    Job Definition

    Now that the container has been defined, OpenShift needs to be told how to use it. This can be done using a CronJob object:

    - apiVersion: batch/v2alpha1
      kind: CronJob
      metadata:
        name: ${JOB_NAME}
        labels:
          job: ${JOB_NAME}
        annotations:
          alpha.image.policy.openshift.io/resolve-names: '*'
      spec:
        schedule: ${CRON_SCHEDULE}
        jobTemplate:             
          spec:
            template:
              metadata:
                labels:          
                  parent: ${JOB_NAME}-parent
              spec:
                containers:
                - name: ${JOB_NAME}
                  image: openshift-clean-slate:latest
                  env:
                  - name: DEPLOYMENT_CONFIGS
                    valueFrom:
                      configMapKeyRef:
                        name: ${JOB_NAME}-config
                        key: deployment.configs              
                restartPolicy: OnFailure

    In this definition, the Job name, the Cron schedule, and the DeploymentConfigs being iterated over in the script are parameterized as part of an OpenShift Template to make consumption easier.

    Service Account Configuration

    The Service Account being used to execute the job must be given Edit rights on the OpenShift Project. This allows it to interact with the DeploymentConfigs, and trigger deployments as if it were a normal user of the platform. This can be done by executing the following command with the OpenShift Command Line Tools in the context of the OpenShift Project the Job needs to be executed within:

    oc policy add-role-to-user edit system:serviceaccount:$(oc project -q):default -n $(oc project -q)

    OpenShift Template

    In order to pull this all together, an OpenShift Template can be created. This exposes a number of parameters:

    • JOB_NAME - A unique identifier for the Job
    • DEPLOYMENT_CONFIGS - A comma-separated list of DeploymentConfigs to target with the Job
    • CRON_SCHEDULE - The standard Cron schedule to execute the Job against

    When instantiated, the template presents an interface that looks like this:

    OpenShift Template for this Blog Post

    The template will create a number of objects in OpenShift:

    • A BuildConfig to create the Job container.
    • The CronJob object is shown above.
    • An ImageStream for the builder image (RHEL Atomic).
    • An ImageStream for the Job container.
    • A ConfigMap containing the targeted DeploymentConfigs, so it can be altered independently between job executions. This takes the value of DEPLOYMENT_CONFIGS parameter and can be edited independently of the job.

    NOTE - It is critical that the lookupPolicy for the Job container's ImageStream is set to 'true' so that the CronJob is able to reference an ImageStream. Without this (and indeed, on OpenShift Container Platform versions earlier than 3.6), a fully qualified reference to the image will need to be used. This is done as part of the template but should be noted for future reference.

    End Result

    So, the template has been created, the Job container has been built. What happens when the Cron schedule fires and the Job are run? Hopefully, it'll be a very quick process, and the Job's logs will need to be examined to get an idea of what's just happened. However, output similar to the following should be displayed:

    Authenticating with token
    Logged into "https://172.30.0.1:443" as "system:serviceaccount:slate:default" using the token provided.
    You have one project on this server: "slate"
    Using project "slate".
    Welcome! See 'oc help' to get started.
    --
    Wiping the slate of kitchensink
    deploymentconfig "kitchensink" rolled out
    Done
    

    Wait. I Could Just Use Jenkins For This

    Absolutely. But that's not what one would traditionally call a lightweight solution to an otherwise simple problem. Achieving the same result in Jenkins involves the integration of jobs and plugins that manage the process of redeploying your containers in OpenShift. An OpenShift Job, by comparison, is simple, repeatable, easy to scale, and uses native functionality rather than relying on a 3rd party orchestrator.

    Resources

    All the resources references in this blog are available on GitHub. Note that a specific tag for this content has been created, as the master branch cannot be guaranteed to be stable.

    https://github.com/benemon/openshift-clean-slate/tree/blog


    To learn more about Red Hat Openshift Container Platform, allowing you to provision, manage, and scale container-based applications.

    Last updated: June 27, 2023

    Recent Posts

    • How to run a fraud detection AI model on RHEL CVMs

    • How we use software provenance at Red Hat

    • Alternatives to creating bootc images from scratch

    • How to update OpenStack Services on OpenShift

    • How to integrate vLLM inference into your macOS and iOS apps

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue