Using RHQ (JON) to monitor Java apps

<p>&nbsp;</p> <quillbot-extension-portal></quillbot-extension-portal>

January 23, 2014
Romain Pelisse
Related topics:
Java
Related products:
Red Hat JBoss Enterprise Application Platform

While there is not yet a RHQ plugin for Glassfish, it is already possible to monitor an instance of Glassfish (GF), using the existing JMX Server resource template. Let's see how this unfolds...

Set up

Foreword: While not difficult, setting this up on a laptop requires running a LOT of Java processes, on top of a database. As the latter needs to be a somewhat older version of PostgreSQL (8.4), you might even end up running it on VM using virt-manager. That is to say, at the end of the day, you do need a little bit of memory and CPU power...

Here what do we need to run exactly:

  • RHQ 4.x (or JON 3.x)
  • PostgreSQL 8.4 (or Oracle)
  • RHQ Agent - downloaded from the server once running
  • Glassfish 3

Once you have RHQ server running, with its agent and the appropriate DB, you'll need to start GF:


$ cd ${GLASSFISH_HOME}
$ ./bin/asadmin start-domain
Waiting for domain1 to start ......
Successfully started the domain : domain1
domain  Location: /home/rpelisse/Products/glassfish3/glassfish/domains/domain1
Log File: /home/rpelisse/Products/glassfish3/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.

Fortunately, you don't have to look for long to get the information you need, as GF kindly print out the information you need (JMX's URL) on its main log:


$ grep -e jmx /home/rpelisse/Products/glassfish3/glassfish/domains/domain1/logs/server.log
[#|2014-01-21T11:57:45.490+0000|INFO|glassfish3.1.2|javax.enterprise.system.jmx.org.glassfish.admin.mbeanserver|_ThreadID=16;_ThreadName=Thread-2;|JMX005: JMXStartupService had Started JMXConnector on JMXService URL service:jmx:rmi://localhost:8686/jndi/rmi://localhost:8686/jmxrmi|#]

To monitor this GF as a simple JMX Server, you just need to click on the platform hosting GF (ie "localhost") and select the folder JMX Servers. From there, you'll find an appropriate where to paste the JMX URL and the define the JVM used. Once this is done, you'll have to wait a little for the agent to connect to the server, and some other "RHQ magic" to happen.

Once this is done, you'll have a new resource, called "Java VM", this is your GF:

However, it does not really get better than this. This crude JMX integration only gave you the data above.

Custom JMX MBeans

It's a shame because if you look into GF's JMX Server using the JConsole, you'll quickly find a lot more information:

Disclaimer: I'm (obviously) no GF's expert - however, while doing this research, I wanted to find an MBean with a somewhat "variable" attributes (database connection, number of session, ...), but I couldn't. I may have overlook something, but it looks like no "live" metrics seem to be exposed through JMX by GF...

Fortunately, RHQ gives us a way to leverage all those extra MBeans. Indeed, before writing a full blown plugin, the project's documentation does mention this possibility. A sample project is provided within RHQ git repository (etc/samples/custom-jmx) and its content is reduce to the bare minimum:


$ tree .
.
|-- pom.xml
`-- src
`-- main
`-- resources
`-- META-INF
`-- rhq-plugin.xml

... So basically, two files, including one of the smallest pom ever:


<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0>"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.custom</groupId>
<artifactId>custom-jmx-plugin</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>RHQ Custom JMX Plugin</name>
<description>Plugin that monitors a set of custom MBeans</description>
</project>

The 'rhq-plugin.xml' descriptor is pretty straightforward and well documented:


<?xml version="1.0" encoding="UTF-8" ?>
<plugin name="GlassfishJmx"
        displayName="Glashfish JMX Plugin"
        version="1.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="urn:xmlns:rhq-plugin"
        xmlns:c="urn:xmlns:rhq-configuration">

   <depends plugin="JMX" useClasses="true"/>
   <service name="First Custom Service"
            description="Our first MBean custom service whose 'id' is static"
            discovery="org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent"
            class="org.rhq.plugins.jmx.MBeanResourceComponent">

      <runs-inside>
         <parent-resource-type name="JMX Server" plugin="JMX"/>
      </runs-inside>

      <plugin-configuration>
         <c:simple-property name="objectName" readOnly="true" default="jmxapp:type=MyJmxObject,id=static"/>
      </plugin-configuration>

      <operation name="toUppercase"
                 displayName="String To Uppercase"
                 description="Converts the given string to all uppercase.">
         <parameters>
            <c:simple-property name="p1" type="string" required="true"/>
         </parameters>
         <results>
            <c:simple-property name="operationResult" type="string"/>
         </results>
      </operation>

      <metric property="RandomNumber"
              displayName="Random Floating Point Number"
              defaultOn="true"
              displayType="summary"
              defaultInterval="30000"
              description="Emits a random number between 0.0 and 1.0"/>
   </service>
</plugin>

With this, you can turn any MBean from GF into a service which means:

  • RHQ can trigger any of the MBean's operation
  • RHQ can gather, as metrics, any of the MBean's attributes.

Regarding the latter with GF, please remember my previous remarks on the lack of "dynamic" values in the GF's MBeans.

Application deployment

Based on the documentation (I didn't find the time to test this), it should be fairly easy with the RHQ Bundle system to deploy and un-deploy applications in GF. Once a new destination on the system (deploiement folder) is defined, a post install task can simply trigger the depoiement of the war now copied over to the host:


${GLASSFISH_HOME}/bin/asadmin deploy /path/to/war

This will requires to wrap 'exec' task on the RHQ Bundle and will definitely not offer a nice error handling, but it should be enough to get started.

A quick conclusion

Overall, without a dedicated plugin for GF, one can already achieved a lot by customizing what is already available through JMX. However, to fully monitor GF, and the application deployed, the amount of custom JMX descriptors to produce might be discouraging. Unless the monitoring need on GF is rather simple, it does not look like a sane strategy to go this way. It would actually be probably easier to realize a dedicated plugin for GF (again IHMO).

On the operation level, while customizing MBeans does provide access to the numerous operations of the already numerous GF's MBeans, crucial operations are missing: start/stop/restart. This means that if RHQ server gets notified that one instance of GF's is not available - i.e. that the JVM running it has crashed, there is nothing it can actually do. While there is probably hooks in the agent or RHQ to implement a simple "run-this-command-on-the-host-to-do-this-operation" on RHQ's agent, the fact remains this will be yet an other work to do, on top of the previous JMX integration.

Another somewhat strong limitation is the small amount of information available on threads by applications deployed. While it may be possible to add one custom MBean by application run by GF, it is still unclear to me if one can easily retrieve such information as:

  • number of session per app, with their size
  • number of (web) threads per app
  • database connection pool usage per app.

From my experience, those are very important metrics to monitor app servers in production.

Down the road, all of this does NOT mean that GF cannot be monitored by RHQ, just that - as always, it mostly depends what you intend to actually to do. If just want RHQ to monitor GF's availability, and maybe check a couple of other things, you probably have enough here already. If you want RHQ to be the central deployment point in an environment featuring hundreds of GF's server, and you want to automatically recover from failed deployment bundled and server's crashes, it is probably a far more sound approach to start working on GF's plugin for JON, especially as it does not look that time consuming and difficult to implement...

Last warning: This is the product of a quick investigation, done by one person, with decent knowledge of RHQ, but none of GF. There is no warranty here that my statements are definitive. I may have missed or overlooked something. Of course, if so, let me know I'll update this entry asap...

Some links:

Last updated: May 31, 2023