Red Hat JBoss Enterprise Application Platform (JBoss EAP) 7 is an open source platform for highly transactional, web-scale Java applications deployed in war/jar/ear and abides by specifications such as Jakarta 8. But it also provides messaging, distributed caching, and clustering features, as explained in the Introduction to JBoss EAP Guide.
In terms of clustering, when JBoss EAP 7 is deployed in Red Hat OpenShift Container Platform (RHOCP), by default JBoss EAP 7 image will have clustering capabilities enabled. To provide those clustering capabilities, JBoss EAP 7 relies on JGroups discovery protocols, such as DNS_PING or KUBE_PING, also known as ping protocols, to provide the ping capabilities for the clustering within the pods.
In this two-part article series, I will provide more information about both of those mechanisms and how they differ. Both articles are based on solutions/articles such as EAP 7 image clustering in OCP 4, which provides extensive details on clustering. In this first part, I will describe the process when using DNS_PING as discovery protocol.
It is important to highlight that although they are enabled by default, clustering capabilities are triggered/started on JBoss EAP 7 after the deployment of clustered applications. Therefore, clustering logs (such as the following one) are logged after the clustering capabilities start:
$ oc logs eap-example-0 | grep ISPN000094
00:01:01,023 INFO [org.infinispan.CLUSTER] (ServerService Thread Pool -- 14) ISPN000094: Received new cluster view for channel ee: [eap-example-0] (1) [eap-example-0]DNS_PING setting
The setting of DNS_PING as a discovery protocol is done using the environment variables JGROUPS_PING_PROTOCOL, OPENSHIFT_DNS_PING_SERVICE_NAME, and OPENSHIFT_DNS_PING_SERVICE_PORT. DNS_PING. The following table summarizes this.
| Environment variable | Purpose |
JGROUPS_PING_PROTOCOL | dns.DNS_PING or Kubernetes KUBE_PING—for this blog post: dns.DNS_PING |
OPENSHIFT_DNS_PING_SERVICE_NAME | Name of the service exposing the ping port on the servers for the DNS discovery mechanism. Example value: eap-app-ping |
OPENSHIFT_DNS_PING_SERVICE_PORT | The port number of the ping port for the DNS discovery mechanism. If not specified, an attempt is made to discover the port number from the SRV records for the service, otherwise the default 8888 is used. |
When doing deployment via template in JBoss EAP 7 (i.e., Source-to-Image deployment is done) the deployment will use DNS_PING instead of KUBE_PING, which is the default usage when deploying via JBoss EAP Operator for example. Also, openshift.DNS_PING is the deprecated name, and the current name is: dns.DNS_PING. You can see other variables here.
DNS_PING operation
The JBoss EAP 7 configuration points to a service via the dns.DNS_PING properties. The following deployment via template:
$ cat /opt/eap/standalone/configuration/standalone-openshift.xml | grep -A 3 -B 3 dns
<stacks>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<protocol type="dns.DNS_PING">
<property name="dns_query">eap-app-ping</property>
<property name="async_discovery_use_separate_thread_per_request">true</property>
</protocol>
<protocol type="MERGE3"/>So then the service eap-app-ping will be used to connect the pods—and to binding those pods a selector is used—this label must be on the pods. For instance, in the following example eap-app is the selector the service will bind, meaning any pod with that selector will be connected via this service.
$ oc get service eap-app-ping -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
description: The JGroups ping port for clustering.
...
spec:
clusterIP: None
clusterIPs:
- None
...
publishNotReadyAddresses: true
selector:
deploymentConfig: eap-app <------------------------ here the deploymentConfig is set for the eap-app
Given JBoss EAP 7 is deployed via DeploymentConfig YAML, so then the DeploymentConfig and the pods will have the respective labels:
$ oc get dc/eap-app -o yaml | grep -B 5 -A 5 JGROUPS
deploymentConfig: eap-app
name: eap-app
spec:
containers:
- env:
- name: JGROUPS_PING_PROTOCOL
value: dns.DNS_PING
- name: OPENSHIFT_DNS_PING_SERVICE_NAME
value: eap-app-ping
- name: OPENSHIFT_DNS_PING_SERVICE_PORT
value: "8888"Pod labels:
selector:
deploymentConfig: eap-app <------------------------ here the deploymentConfig is set for the eap-appThe pod's selector is crucial for the service (svc) to find it—otherwise, the service will not find the pods and therefore JBoss EAP 7 won't be able to cluster.
Service details
Clustering capabilities rely solely on eap-app-ping (as previously shown), so other services are not mandatory. The eap-app-ping is a headless service, meaning the service will point straight to the Pods' IPs. This can be shown as:
oc get ep,svc -o wide
NAME ENDPOINTS AGE
endpoints/eap-app-ping 1.1.1.127:8888,1.1.1.128:8888 75mIn the previous endpoints, 1.1.1.127:8888, 1.1.1.128:8888 (*example IPs) are two pods IPs. So, the requests go straight to those pods. This can be used to confirm if the headless service is functioning properly and connecting the pods.
From the OpenShift Container Platform console, go to Network > Services, and the eap-app-ping should show the related pods (the ones it connects to) on the console, this will show more details on this service.
Troubleshooting DNS_PING
One can set DNS_PING for tracing or JGroups and have more details on the clustering operation:
### dns_ping as trace
$ oc set env dc/eap-app LOGGER_CATEGORIES=org.jgroups.protocols.DNS_PING:TRACE
### jgroups as trace
$ oc set env dc/eap-app LOGGER_CATEGORIES=org.jgroups:TRACEThe tracing log will be as follows:
-
dns.DNS_PING: set the discovery protocol. logging: Category org.jgroups.protocols.dnswith level TRACE. This message will be added in case TRACE is set.- Initializing DNS Context with factory:
com.sun.jndi.dns.DnsContextFactoryand URL. - Resolving DNS query:
eap-app-pingof a type: A. The query is sent to eap-app-ping service. - Collect the entries:
eap-app-3-gkcjb: entries collected from DNS (in 2 ms): [127.1.1.1:0, 127.1.1.2:0, 127.1.1.3:0, 127.1.1.4:0]. Collect these. - Send discovery request: Sending discovery request to
127.1.1.2.:7600. Example request sent. - Received:
GET_MBRS_REQandGET_MBRS_RSPfor each member. - The node will get
GET_MBRS_RSPfrom other nodes, which will join the cluster:ISPN100000: Nodeeap-app-pod-2joined the cluster. - The reply will be name address and coordinator: sending the response
eap-app-pod-1,name=eap-app-pod-1,addr=127.1.1.:7600, coordinator. - Rebalance:
ISPN100002: Starting rebalance with members [eap-app-pod-1,eap-app-pod-2], phaseREAD_OLD_WRITE_ALL, topology id 2.
A note about JBoss EAP 8
Regarding JBoss EAP 8 container deployment with clustering, the default simple build (such as the following) will not bring clustering capabilities:
### Value file below does not have clustering capabilities:
build:
uri: https://github.com/jboss-developer/jboss-eap-quickstarts.git
ref: 8.0.x
contextDir: helloworld
env:
- name: MAVEN_OPTS_APPEND
value: '-XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=256m'
deploy:
replicas: 3To have clustering capabilities, deploy JBoss EAP 8 with a configuration/layer that provides JGroups subsystem, such as cloud-default-config, which is a layer that mimics what used to be present in the JBoss EAP 7.4.x images default installed server.
build:
uri: https://github.com/jboss-developer/jboss-eap-quickstarts.git
ref: 8.0.x
contextDir: helloworld
env:
- name: GALLEON_PROVISION_FEATURE_PACKS
value: 'org.jboss.eap:wildfly-ee-galleon-pack, org.jboss.eap.cloud:eap-cloud-galleon-pack'
- name: GALLEON_PROVISION_LAYERS
value: 'cloud-default-config'
- name: GALLEON_PROVISION_CHANNELS
value: 'org.jboss.eap.channels:eap-8.0'
deploy:
replicas: 3 To verify clustering capabilities see the start of the JBoss EAP pods, see the following.
Disabled:
INFO Clustering feature is not enabled, no jgroups subsystem present in server configuration.Enabled:
INFO Configuring JGroups cluster traffic encryption protocol to SYM_ENCRYPT.
WARN Detected missing JGroups encryption configuration, the communication within the cluster WILL NOT be encrypted.
WARN No password defined for JGroups cluster. AUTH protocol will be disabled. Please define JGROUPS_CLUSTER_PASSWORD.
INFO Configuring JGroups discovery protocol to dns.DNS_PING OpenShift Service Mesh Usage and Support
Red Hat OpenShift Service Mesh is not currently supported in JBoss EAP 7 nor in its side-car or ambient mesh configurations. However, evidently, in some circumstances, it can be useful for testing/verifying observability or security as well as access control. Red Hat Service Mesh Operator OpenShift Container Image uses Istio and allows the insertion of the envoy proxy as a sidecar (so two containers in the same pod: JBoss EAP and envoy proxy). The requests and connection exchanges will be done through the envoy.
Warning:
Be aware of peer authentication, which is set on the envoy sidecar that has implications on JBoss EAP 7 clustering, even halting cluster capabilities in the case of mtls strict.
What's next
This article described the setting up and operation of JBoss EAP 7 clustering via DNS_PING. The solutions EAP 7 image clustering in OCP 4, EAP 7 clustering deployments boundaries in OCP cluster, and EAP 7 DNS_PING tracing logs in OCP 4 describe the JBoss EAP 7 clustering properties and setting, boundaries of the JBoss EAP 7 DNS_PING services, and DNS_PING tracing usage and output interpretation.
The next installment will address the usage and operation of KUBE_PING as discovery protocol, which uses DNS A or SRV record entries to perform discovery, and it was done specifically for OpenShift/Kubernetes (referenced here).
For any other specific inquiries, please open a case with Red Hat support. Our global team of experts can help you with any issues. Special thanks to Mr. Dennis Reed for his outstanding collaboration and Alexander Barbosa for his review of this article.
