Automate workshop setup with Ansible playbooks and CodeReady Workspaces

At Red Hat, we do many in-person and virtual workshops for customers, partners, and other open source developers. In most cases, the workshops are of the "bring your own device" variety, so we face a range of hardware and software setups and corporate endpoint-protection schemes, as well as different levels of system knowledge.

In the past few years, we've made heavy use of Red Hat CodeReady Workspaces (CRW). Based on Eclipse Che, CodeReady Workspaces is an in-browser IDE that is familiar to most developers and requires no pre-installation or knowledge of system internals. You only need a browser and your brain to get hands-on with this tech.

We've also built a set of playbooks for Red Hat Ansible to automate our Quarkus workshop. While they are useful, the playbooks are especially helpful for automating at-scale deployments of CodeReady Workspaces for Quarkus development on Kubernetes. In this article, I introduce our playbooks and show you how to use them for your own automation efforts.

Automating at scale: An overview

While we use CodeReady Workspaces and the Ansible playbooks introduced in this article to automate our Quarkus workshop, many companies use CodeReady Workspaces to automate the onboarding of new developers at scale. In that case, using CodeReady Workspaces also helps to protect corporate intellectual property (that is, source code) and minimize the "works on my machine" excuse for bugs.

Regardless of whether you are running a workshop or an onboarding process, making the experience as smooth as possible requires considerable setup. For a workshop, you need to deploy Red Hat OpenShift, scale it to meet the demands of the number of users expected, and install and configure CodeReady Workspaces for every user. For the best experience, you should also "pre-warm" each workspace so that it is already running by the time you are done with your intro slides. You will also need to install any Operators that you will use as part of the workshop.

In the next sections, I'll go through each of these steps and the Ansible playbooks that we've built to automate them. Most of the setup can be applied to creating custom stacks that follow both company policy and IT policy, and also meet developers' needs.

Installing OpenShift

OpenShift is used for hybrid cloud infrastructure, so for at-scale deployments, you aren't "installing" it in the classic sense of downloading a zip file, unzipping it, and running it on your desktop. You can do that type of install with CodeReady Containers, but running locally is just not an option when you are supporting tens or hundreds of developers in a workshop.

You can easily provision OpenShift on several different public and private clouds. While deploying OpenShift is out of scope for this article, we found it useful to deploy an extra OpenShift worker node for every five students, where each node has 64GiB of memory. That setup supports a positive workshop experience for every student.

For the Quarkus Workshop, we have students doing native Quarkus builds, which require extra memory. Each student also deploys their own Kafka clusters and a few other items. So, we just run through the workshop once, leave everything running, and then add everything up to determine the amount of memory needed per user. Keep in mind that for CodeReady Workspaces, we use the "per-workspace" persistent volume claims (PVC) strategy, where each workspace (and therefore each user) gets its own storage. If you choose to follow that strategy, you will need to ensure that you have enough storage space. The more CPU you can afford, the better.

Once you have OpenShift installed, you will need to create users. You can use basic Linux shell scripting and the oc CLI to override the default OpenShift authentication mechanism and supply an htpasswd file containing your users (including an admin user). You will also need the htpasswd utility and the yq utility (version 3 or higher) for this Bash script:

#!/bin/bash
NUMUSERS=20
TMPHTPASS=$(mktemp)
for i in {1..$NUMUSERS} ; do
    htpasswd -b ${TMPHTPASS} "user$i" 'somepassword'
done

htpasswd -b ${TMPHTPASS} admin 'adminpassword'

$ oc -n openshift-config delete secret workshop-user-secret

$ oc -n openshift-config create secret generic workshop-user-secret --from-file=htpasswd=${TMPHTPASS}

$ oc -n openshift-config get oauth cluster -o yaml | \
  yq d - spec.identityProviders | \
  yq w - -s htpass-template.yaml | \
  oc apply -f -

sleep 20 # don't shoot the messenger, Operators are "eventually consistent"

$ oc adm policy add-cluster-role-to-user cluster-admin admin

The htpass-template.yaml template used with yq (version 3) looks like:

spec.identityProviders[+]:
  name: htpassidp
  type: HTPasswd
  mappingMethod: claim
  htpasswd:
    fileData:
      name: workshop-user-secret

Running the script with this template merges a new identity provider into the OpenShift auth flow so that users can log in. You could also use Ansible to set up this authorization process, but I haven't yet found the time to convert it.

Deploying CodeReady Workspaces

We use the CodeReady Workspaces Operator for this installation. To automate the installation, we use a bit of Ansible in an Ansible playbook. If the namespace does not already exist, we use the k8s module to create one, along with the OperatorGroup and Subscription (idempotency and all):

# create codeready namespace
- name: create codeready namespace
  k8s:
    state: present
    kind: Project
    api_version: project.openshift.io/v1
    definition:
      metadata:
        name: "codeready"
        annotations:
          openshift.io/description: ""
          openshift.io/display-name: "CodeReady Project"

# deploy codeready operator
- name: Create operator subscription for CodeReady
  k8s:
    state: present
    merge_type:
    - strategic-merge
    - merge
    definition: "{{ lookup('file', item ) | from_yaml }}"
  loop:
  - ./files/codeready_operatorgroup.yaml
  - ./files/codeready_subscription.yaml

# wait for CRD to be a thing
- name: Wait for CodeReady CRD to be ready
  k8s_facts:
    api_version: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    name: checlusters.org.eclipse.che
  register: r_codeready_crd
  retries: 200
  delay: 10
  until: r_codeready_crd.resources | list | length == 1

# deploy codeready CR
- name: Create CR for CodeReady
  k8s:
    state: present
    merge_type:
    - strategic-merge
    - merge
    definition: "{{ lookup('file', item ) | from_yaml }}"
  loop:
  - ./files/codeready_cr.yaml

# wait for CodeReady to be up
- name: wait for CRW to be running
  uri:
    url: https://codeready-codeready.{{ route_subdomain }}/dashboard/
    validate_certs: false
  register: result
  until: result.status == 200
  retries: "120"
  delay: "15"

The bits of code that are waiting on the custom resource definition (CRD) are important: If you try to create a custom resource (CR) based on a CRD before the CRD is known to the system, it will fail. Furthermore, it takes non-zero time to gain that knowledge once the Operator is installed.

At the end, we also use the uri module to wait for CodeReady Workspaces itself, as we do some additional configuration next.

OperatorGroup

The OperatorGroup is defined in codeready_operatorgroup.yaml. It is pretty simple, but it's required for Operators to be able to, well, operate:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  generateName: codeready-
  annotations:
    olm.providedAPIs: CheCluster.v1.org.eclipse.che
  name: codeready-operator-group
  namespace: codeready
spec:
  targetNamespaces:
    - codeready

Subscription

The Subscription in codeready_subscription.yaml is also basic:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: codeready-workspaces
  namespace: codeready
spec:
  channel: latest
  installPlanApproval: Automatic
  name: codeready-workspaces
  source: redhat-operators
  sourceNamespace: openshift-marketplace

CheCluster object

Finally, once the Operator registers its CRDs in Kube, we can create the CheCluster object in codeready_cr.yaml. Creating the CheCluster kicks off the install:

apiVersion: org.eclipse.che/v1
kind: CheCluster
metadata:
  name: codeready-workspaces
  namespace: codeready
spec:
  server:
    cheImageTag: ''
    cheFlavor: codeready
    devfileRegistryImage: ''
    pluginRegistryImage: ''
    tlsSupport: true
    selfSignedCert: false
    serverMemoryRequest: '2Gi'
    serverMemoryLimit: '6Gi'
    customCheProperties:
      CHE_LIMITS_WORKSPACE_IDLE_TIMEOUT: "0"
  database:
    externalDb: false
    chePostgresHostName: ''
    chePostgresPort: ''
    chePostgresUser: ''
    chePostgresPassword: ''
    chePostgresDb: ''
  auth:
    openShiftoAuth: false
    identityProviderImage: ''
    externalIdentityProvider: false
    identityProviderURL: ''
    identityProviderRealm: ''
    identityProviderClientId: ''
  storage:
    pvcStrategy: per-workspace
    pvcClaimSize: 1Gi
    preCreateSubPaths: true

Note the memory limits, which are tuned for the containers in our custom CodeReady Workspaces stack. We also set the CHE_LIMITS_WORKSPACE_IDLE_TIMEOUT here. It is rather annoying to walk away for a short time and find that your lab has timed out and needs a refresh (or requires you to log in again) when you return. Of course, neither of these settings should be used in production.

Tuning Keycloak

It is not possible to use OpenShift's built-in authentication mechanism to pre-create and pre-start workspaces. Doing that would require each user to log in to OpenShift and link the user's account details to Red Hat Single Sign-O. (That, by the way, is why you see openShiftoAuth: false in the CheCluster resource.)

The workaround to this issue is to create the same set of users in CodeReady Workspaces, again using Ansible:

- name: create codeready users
  include_tasks: add_che_user.yaml
  vars:
    user: "{{ item }}"
  with_list: "{{ users }}"

In this example, users is just an array of usernames in Ansible (for instance, [user1, user2, ...]). We loop through and add the user in add_che_user.yaml, which uses the CodeReady Workspaces REST API to get credentials for the SSO admin user and create the users:

- name: Get codeready SSO admin token
  uri:
    url: https://keycloak-codeready.{{ route_subdomain }}/auth/realms/master/protocol/openid-connect/token
    validate_certs: false
    method: POST
    body:
      username: "{{ codeready_sso_admin_username }}"
      password: "{{ codeready_sso_admin_password }}"
      grant_type: "password"
      client_id: "admin-cli"
    body_format: form-urlencoded
    status_code: 200,201,204
  register: codeready_sso_admin_token

- name: Add user {{ user }} to Che
  uri:
    url: https://keycloak-codeready.{{ route_subdomain }}/auth/admin/realms/codeready/users
    validate_certs: false
    method: POST
    headers:
      Content-Type: application/json
      Authorization: "Bearer {{ codeready_sso_admin_token.json.access_token }}"
    body:
      username: "{{ user }}"
      enabled: true
      emailVerified: true
      firstName: "{{ user }}"
      lastName: Developer
      email: "{{ user }}@no-reply.com"
      credentials:
        - type: password
          value: "{{ workshop_che_user_password }}"
          temporary: false
    body_format: json
    status_code: 201,409

This playbook has a few variables:

• route_subdomain is the default OpenShift subdomain for your cluster (use oc whoami --show-cluster to discover the cluster).
• workshop_che_user_password is your user's desired password.
• codeready_sso_admin_username/codeready_sso_admin_password is the admin username and password for the Keycloak instance used by CodeReady Workspaces.

To programmatically discover the Keycloak admin username and password from the deployed Keycloak's environment variables, you can use a little more Ansible code and the k8s_facts module:

- name: Get codeready keycloak deployment
  k8s_facts:
    kind: Deployment
    namespace: codeready
    name: keycloak
  register: r_keycloak_deployment

- name: set codeready username fact
  set_fact:
    codeready_sso_admin_username: "{{ r_keycloak_deployment.resources[0].spec.template.spec.containers[0].env | selectattr('name','equalto','SSO_ADMIN_USERNAME') |map (attribute='value') | list | first }}"

- name: set codeready password fact
  set_fact:
    codeready_sso_admin_password: "{{ r_keycloak_deployment.resources[0].spec.template.spec.containers[0].env | selectattr('name','equalto','SSO_ADMIN_PASSWORD') |map (attribute='value') | list | first }}"

Next up, we increase the SSO token expiration and SSO session timeout (again, this lets us avoid irritating logouts during a workshop):

- name: Increase codeready access token lifespans
  uri:
    url: https://keycloak-codeready.{{ route_subdomain }}/auth/admin/realms/codeready
    validate_certs: false
    method: PUT
    headers:
      Content-Type: application/json
      Authorization: "Bearer {{ codeready_sso_admin_token.json.access_token }}"
    body:
      accessTokenLifespan: 28800
      accessTokenLifespanForImplicitFlow: 28800
      actionTokenGeneratedByUserLifespan: 28800
      ssoSessionIdleTimeout: 28800
      ssoSessionMaxLifespan: 28800
    body_format: json
    status_code: 204

Pre-warming user workspaces

Finally, we're ready to pre-create and pre-warm the CRW workspaces:

- name: Pre-create and warm user workspaces
  include_tasks: create_che_workspace.yaml
  vars:
    user: "{{ item }}"
  with_list: "{{ users }}"

We will repeat a loop similar to what we did to create the user workspaces. This time, we call create_che_workspace.yaml, which uses the CodeReady Workspaces REST API:

- name: "Get Che {{ user }} token"
  uri:
    url: https://keycloak-codeready.{{ route_subdomain }}/auth/realms/codeready/protocol/openid-connect/token
    validate_certs: false
    method: POST
    body:
      username: "{{ user }}"
      password: "{{ workshop_che_user_password }}"
      grant_type: "password"
      client_id: "admin-cli"
    body_format: form-urlencoded
    status_code: 200
  register: user_token

- name: Create workspace for {{ user }} from devfile
  uri:
    url: "https://codeready-codeready.{{ route_subdomain }}/api/workspace/devfile?start-after-create=true&namespace={{ user }}"
    validate_certs: false
    method: POST
    headers:
      Content-Type: application/json
      Authorization: "Bearer {{ user_token.json.access_token }}"
    body: "{{ lookup('template', './templates/devfile.json.j2') }}"
    body_format: json
    status_code: 201,409
  register: workspace_def

About the devfile

If you are wondering about the devfile.json.j2, it is an Ansible Jinja2 template of a CodeReady devfile.

You can find the devfile for this example here. The interesting parts are:

  "components": [
    {
      "id": "redhat/quarkus-java11/latest",
      "type": "chePlugin"
    },

Note that the devfile includes the Quarkus plugin for the workspace, which provides IDE features like autocompletion and other tidbits:

      "image": "image-registry.openshift-image-registry.svc:5000/openshift/quarkus-stack:2.1",

Here, we reference a Che stack that has been pre-generated and deployed into OpenShift as an ImageStream using:

apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  name: quarkus-stack
  namespace: openshift
spec:
  tags:
  - annotations:
      description: Quarkus stack for Java and CodeReady Workspaces
      iconClass: icon-java
      supports: java
      tags: builder,java
      version: "2.1"
    from:
      kind: DockerImage
      name: quay.io/openshiftlabs/quarkus-workshop-stack:2.1
    name: "2.1"

We built the stack using a Dockerfile that also includes utilities (oc, kn, tkn, and GraalVM). It runs a couple of test builds to pre-populate the Maven .m2 repository in the image so that users don't download the Internet every time they start the workshop. By pre-pulling this image into OpenShift, we significantly reduce workspace startup time. There is also the Image Puller, which I have not yet used. It looks promising for eliminating some of this logic.

Conclusion

In summary, automating CodeReady Workspace deployments at scale can significantly improve how students experience your workshops. Doing as much as possible up front lets students get to the learning, without waiting for installations, warm up, and so on.

This article introduced some of the Ansible playbooks we've created to automate and improve user experiences with our workshops. Additional options include:

• Deploying other Operators (Strimzi, Jaeger, and so on).
• Creating custom Keycloak realms for the workshop.
• Verifying other components of the workshop are correctly deployed.

Have a look at the Deploy Quarkus Workshop into an OpenShift 4 Cluster playbook. There might be other bits that you can use! Also, if you're interested in the onboarding new developers example, check out the article CodeReady Workspaces Delivers Kubernetes-Native IDE.