Overview: Cross site and cross applications with Red Hat OpenShift and Red Hat Data Grid

One of the biggest problems in distributed applications is data replication. This task becomes even more complex when different technologies are involved, and even more so when we are dealing with applications distributed in different clusters. In this learning path, we will demonstrate how an application developed in three different technologies (Quarkus, .NET Core 7, and Golang) share data with each other through the Red Hat Data Grid and are distributed in two different Red Hat OpenShift clusters, thus demonstrating the power of the functionality cross-site replication offered by Data Grid.

The environment

We will work with two independent OpenShift clusters, site-1 and site-2. In each OpenShift cluster, we will use the facilities provided by the Operators to deploy and configure Data Grid and cache instances.

In this learning path created by Marcelo Daniel Sales, we will not go into detail about each of the tools used. However, there is a vast collection of documents and articles about OpenShift and Data Grid available at developers.redhat.com.

Create the project on both sites

Create a new project:  name _rhdg-xsite_ (Figure 1).

Generate certificates and Identities

First, we need to generate some identities files, certificates, and a key pair. Data Grid uses these artifacts for communication between clients and distributed clusters. For this demo, we generated certificates using the following script:

#!/bin/bash

set -e
set -x

WORK_DIR="$PWD"
CA_CRT="${WORK_DIR}/ca.crt"
CA_KEY="${WORK_DIR}/ca.key"
CA_KEYSTORE="${WORK_DIR}/ca-keystore.p12"

CA_PASS="secretCaPassword"
CA_KEYSTORE_PASS="caSecret"

KEYSTORE_PASS="secret"

C="ES"
S="Valencia"
L="Valencia"
O="infinispan.org"
OU="infinispan-server"
CN="dg"

function fail() {
    echo $@
    exit 1
}

function failIfFileExists() {
    [ -f "${1}" ] && fail "${1} already exists"
}

function generate_ca_keystore() {
    echo "Generating CA. cert=${CA_CRT} key=${CA_KEY}"
    if [[ -f "${CA_CRT}" && -f "${CA_KEY}" ]] ; then
        echo "Certificate and key already exist. Skipping"
    else
        openssl req -new -x509 -keyout ${CA_KEY} -out ${CA_CRT} -passout pass:${CA_PASS} -subj "/C=${C}/ST=${S}/L=${L}/O=${O}/CN=${CN}"
    fi
    if [ -f "${CA_KEYSTORE}" ]; then
        echo "CA keystore already exist. Skipping"
    else
        keytool -keystore ${CA_KEYSTORE} -alias CARoot -import -file ${CA_CRT} -storepass ${CA_KEYSTORE_PASS} -noprompt -trustcacerts -storetype pkcs12
    fi
}

function generate_signed_certificate() {
    local alias=$1
    local crt="${WORK_DIR}/${alias}.crt"
    local signed_crt="${WORK_DIR}/${alias}-signed.crt"
    local keystore="${WORK_DIR}/${alias}-keystore.p12"
    local keystore_pass="${KEYSTORE_PASS}"

    if [ ! -f "${crt}" ] ; then
        echo "Generating certificate ${crt}"
        keytool -genkey -alias ${alias} -keyalg RSA -keystore ${keystore} -keysize 2048 -storetype pkcs12 -storepass ${keystore_pass} -noprompt -dname "CN=${alias}, OU=${OU}, O=${O}, L=${L}, S=${S}, C=${C}"
        keytool -keystore ${keystore} -alias ${alias} -certreq -file "${crt}" -storepass ${keystore_pass} -storetype pkcs12
    fi

    if [ ! -f "${signed_crt}" ] ;  then
        echo "Sign server certificate. ca_cert=${CA_CRT} ca_key=${CA_KEY} server_cert=${crt} signed_cert=${signed_crt}"
        openssl x509 -req -CA ${CA_CRT} -CAkey ${CA_KEY} -in "${crt}" -out "${signed_crt}" -days 365 -CAcreateserial -passin pass:${CA_PASS}
    fi

    echo "Import signed certificate"
    keytool -keystore ${keystore} -alias CARoot -import -file ${CA_CRT} -storepass ${keystore_pass} -noprompt -trustcacerts -storetype pkcs12
    keytool -keystore ${keystore} -alias ${alias} -import -file ${signed_crt} -storepass ${keystore_pass} -storetype pkcs12
}

generate_ca_keystore
generate_signed_certificate dg

Save the above content in a generate_certs.sh file and then run it to generate the certificates. After generating the certificates, we will create secrets on both sites using these same certificates which will be used later by Data Grid.

Let's use the OpenShift Client to create the secrets.

1. Log into site-1 and copy the authentication token by clicking on the username in the upper-right corner and selecting Copy login command. 
2. Select Display token and copy the line that starts with "oc login --token=sha256".

Create secrets in OpenShift

Run the commands below to create the secrets using the previously generated keys, using the correct project (-n rhdg-xsite in our example) and the correct files (dg-keystore.p12 and ca-keystore.p12):

oc -n rhdg-xsite create secret generic xsite-keystore "--from-file=keystore.p12=$(pwd)/dg-keystore.p12" "--from-literal=password=secret" "--from-literal=type=pkcs12"
oc -n rhdg-xsite create secret generic xsite-truststore "--from-file=truststore.p12=$(pwd)/ca-keystore.p12" "--from-literal=password=caSecret"

Now let's create a secret with identities that will be used to authenticate users who will access the Data Grid. We can use the file structure below as an example:

identities.yaml

credentials:
- username: admin
  password: password
  roles:
  - admin
- username: developer
  password: password
  roles:
  - admin
- username: operator
  password: password
  roles:
  - deployer
- username: user1
  password: password
  roles:
  - monitor

oc create secret -n rhdg-xsite generic --from-file=identities.yaml dg-identities-secret

Repeat the steps above, creating the secrets in Red Hat OpenShift, but this time, point to site-2.

1. Log into site-2, then copy and paste the command "oc login.." with the authentication token from site-2.

Note: Do not generate other certificates. Use the same certificates for site-2 that you created for site-1. At the end of these steps, both sites should have the following secrets:

• dg-identities-secret
• xsite-keystore
• xsite-truststore

Data Grid installation and configuration on site-1

1. Access the administration menu in the upper-left corner (Figure 2).

   

2. Select Operators, then Operator Hub.
3. Find and install the Data Grid operator (Figure 3).

   

4. After the installation, select the operator in Installed Operators (Figure 4).

   

5. Select the Infinispan Cluster tab, then Create Infinispan. Note: Red Hat Data Grid is a distributed, in-memory, data-store built from a tried and trusted open-source technology called Infinispan.
6. Select configure via: yaml view and use the content below as an example:

   apiVersion: infinispan.org/v1
   kind: Infinispan
   metadata:
     name: dg
     namespace: rhdg-xsite
   spec:
     security:
       endpointAuthentication: true
       endpointSecretName: dg-identities-secret
     container:
       memory: 1Gi
     expose:
       type: Route
     service:
       container:
         storage: 1Gi
       sites:
         local:
           discovery:
             launchGossipRouter: true
             type: gossiprouter
           encryption:
             routerKeyStore:
               alias: dg
               secretName: xsite-keystore
             transportKeyStore:
               alias: dg
               secretName: xsite-keystore
             trustStore:
               filename: truststore.p12
               secretName: xsite-truststore
           expose:
             type: Route
           maxRelayNodes: 1
           name: site-1
         locations:
           - name: site-2
       type: DataGrid

7. Click Create.

For now, that's all we need to do on site-1. Let's perform the configuration of site-2.

Data Grid installation and configuration on site-2

Repeat steps 1 - 5 as performed during the installation and configuration of site-1.

1. Select configure via: yaml view and use the content below as an example:

   apiVersion: infinispan.org/v1
   kind: Infinispan
   metadata:
     name: dg
   spec:
     security:
       endpointSecretName: dg-identities-secret
     container:
       memory: 1Gi
     expose:
       type: Route
     service:
       container:
         storage: 1Gi
       sites:
         local:
           discovery:
             launchGossipRouter: false
             type: gossiprouter
           encryption:
             routerKeyStore:
               alias: dg
               secretName: xsite-keystore
             transportKeyStore:
               alias: dg
               secretName: xsite-keystore
             trustStore:
               filename: truststore.p12
               secretName: xsite-truststore
           expose:
             type: Route
           maxRelayNodes: 1
           name: site-2
         locations:
           - name: site-1
             url: infinispan+xsite://<DATAGRID-ROUTE>-<SITE1-PROJECT>.<SITE1-DOMAIN>:443
       type: DataGrid
     configListener:
       enabled: true
     replicas: 1

2. Substitute the values between < and >. To complete this field, you must look at site-1 for the dg-router container. Find and use the route value and replace it in this field, removing the protocol (Figure 5).

   

3. For example: If the value of the route configured in the dg-router is something like:

    https://dg-route-site-rhdg-xsite.apps.ocp4.mydomain.com

4. Then the value for url field will be:

   infinispan+xsite://dg-route-site-rhdg-xsite.apps.ocp4.mydomain.com:443

5.  Select Create.

At this point, we have a cluster configured between the two OpenShift instances. Look at the Infinispan status on the operator's page, and make sure the CrossSiteViewFormed status is present (Figure 6).

You can find more information regarding Red Hat Data Grid cross-site replication here.