JBoss Fabric Zookeeper Troubleshooting

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.

JBoss Fuse Fabric uses this Zookeeper maintaining its configuration so that it could be accessible to all the containers in the fabric environment. Let's look at some of the some of the issues we face or may face with Zookeeper in Fabric. Fabric container not detecting changes is done in Fabric Ensemble.

In your Fabric Ensemble, you can come across situations where you need to move the container to some other machine. Even if you move the server to another location, still the Zookeeper configuration needs to be updated with the latest hostname of the machine. One way of doing so is to add the previous hostname in the /etc/hosts something like shown below:

127.0.0.1  server0   server1

If the Zookeeper URL does not point to the updated or correct hostname, you can do the following steps:

• Run the command on Karaf console to check the current Zookeeper URL it is pointing to.
  

  JBossFuse:karaf@root> config:list |grep zookeeper.url
      zookeeper.url = server0:2181

• Run the following commands to update the Zookeeper URL.
  

  JBossFuse:karaf@root> config:edit io.fabric8.zookeeper
  JBossFuse:karaf@root> config:proplist service.pid = io.fabric8.zookeeper
      zookeeper.password = ZKENC=YWRtaW4=
      zookeeper.url = server0:2181
      fabric.zookeeper.pid = io.fabric8.zookeeper
  JBossFuse:karaf@root> config:propset zookeeper.url server1:2181
  JBossFuse:karaf@root> config:update

• Once you run the above command you can check to see the Zookeeper URL is updated.

It is sometimes seen that a fabric container behaves as it is not part of fabric anymore. In addition to this, we face an issue that we see that the process is running but no logs are written in the container. You'll see an exception as:

14:22:12,883 | INFO | .aoins.com:2182) | ClientCnxn | 57 - io.fabric8.fabric-zookeeper - 1.0.0.redhat-412 | Opening socket connection to server HOSTNAME1/10.17.56.148:2182. Will not attempt to authenticate using SASL (unknown error)
14:22:12,889 | INFO | .aoins.com:2182) | ClientCnxn | 57 - io.fabric8.fabric-zookeeper - 1.0.0.redhat-412 | Socket connection established to HOSTNAME1/101.17.56.148:2182, initiating session
14:22:12,960 | WARN | erSender[myid=4] | QuorumCnxManager | 57 - io.fabric8.fabric-zookeeper - 1.0.0.redhat-412 | Cannot open channel to 5 at election address HOSTNAME2/10.17.56.150:2183
java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)[:1.7.0_71]
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)[:1.7.0_71]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)[:1.7.0_71]

This means the Zookeeper configuration is having a discrepancy with the number of containers running. To troubleshoot you need to compare the Zookeeper configuration with a running container and find the discrepancy and fix as above.

You can look for the zookeeper config file in the bundle cache at location:
Go to location ${karaf.home}/data/cache search for the zookeeper.config file.

Run command:

$ find . -name zookeeper.config

$ find . -name zookeeper.config

    bundle8/data/config/io/fabric8/zookeeper.config

$ less bundle7/data/config/io/fabric8/zookeeper.config

    zookeeper.url="HOSTNAME1:2182,HOSTNAME2:2183,HOSTNAME3:2184"

The number of containers should equal to the number to comma-separated values in zookeeper.url. You can perform the steps mentioned at the start of the blog to update.

Click here to for an overview of Red Hat JBoss Fuse a lightweight and modular integration platform.