Quiver CC0

As part of the Red Hat UKI Professional Services team, I have worked with several customers who are implementing AMQ Broker on Red Hat OpenShift Container Platform (OCP). One question customers typically ask is, “How do we validate that the AMQ configuration is correct for our scenario?” Previously, I would have suggested one of the following:

These tools can give you indicators around:

  • Is the broker up and running? That is, can it receive/publish messages for this configuration?
  • Can the broker handle a certain performance characteristic? That is, what is my minimum publish rate per second for this configuration?
  • And much more.

The problem with these tools is that you cannot choose the client technology. This could lead to real-world differences and limited technology choices, which in turn might lead you down the wrong technology path. In other words:

  • Do you get the same performance from JMeter versus the AMQ clients you would use in production? Are you comparing like for like? Apples with apples?

So, what do I think is the answer? Quiver [1]. In this article, I'll provide an overview and demo of using Quiver with Red Hat AMQ on Red Hat OpenShift.  If you're looking for more information on Red Hat AMQ and how it can help, check out this webinar.

What is Quiver?

Straight from the Quiver GitHub page:

Quiver implementations are native clients (and sometimes also servers) in various languages and APIs that either send or receive messages and write raw information about the transfers to standard output. They are deliberately simple.

The quiver-arrow command runs a single implementation in send or receive mode and captures its output. It has options for defining the execution parameters, selecting the implementation, and reporting statistics.

The quiver command launches a pair of quiver-arrow instances, one sender and one receiver, and produces a summary of the end-to-end transmission of messages.

In my own words;

Quiver is a tool that starts both a publisher and receiver client with the aim of completing as fast as possible. Once the publisher and receiver have completed; a statistical output is generated which can give you insight into how the test has performed.

The clients are based on the Apache Qpid project that offers; Java, C++, Python and Javascript implementations to name a few. The clients primarily target JMS and AMQP users.

Recorded demo

Feeling lazy and want to watch an asciinema recording?

The demo will show the following:

  • AMQ Broker and AMQ Interconnect being deployed onto OKD running on Minishift
  • Send and receive messages via Quiver using the Core protocol to the broker
  • Send and receive messages via Quiver using a JMS client over AMQP protocol to the broker
  • Send and receive messages via Quiver using a JMS client over AMQP protocol to interconnect

asciicast

Do it yourself demo

Want to play with Quiver, AMQ, and OCP yourself?

The demo uses an OCP template which requires two arguments:

  • DOCKER_IMAGE; location of the fully qualified docker pull URL. This can be resolved via the imported image stream; oc get is quiver -o jsonpath=‘{.status.dockerImageRepository}’
  • DOCKER_CMD; the quiver command, in JSON array format, you want to execute.

Demo prerequisites

It is presumed that you have the following:

Setup

  1. Create project
     $ oc new-project quiver
  2. Import the Quiver image
     $ oc import-image quiver:latest --from=docker.io/ssorj/quiver --confirm
  3. Add the view role to the default service account
     $ oc policy add-role-to-user view system:serviceaccount:$(oc project -q):default
  4. Deploy AMQ Broker
     $ oc new-app --template=amq-broker-72-basic \
        -e AMQ_PROTOCOL=openwire,amqp,stomp,mqtt \
        -e AMQ_QUEUES=quiver \
        -e AMQ_ADDRESSES=quiver \
        -e AMQ_USER=anonymous \
        -e ADMIN_PASSWORD=password
    
     $ oc get pods --watch
  5. Deploy AMQ Interconnect
     $ oc new-app --template=amq-interconnect-1-basic
    
     $ oc get pods --watch

Send messages to AMQ Broker

  1. Send 1,000,000 messages to the broker via the core protocol
     $ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
             DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
             DOCKER_CMD="[\"quiver\", \"//broker-amq-tcp:61616/activemq-artemis-jms\", \"--impl\", \"activemq-artemis-jms\", \"--verbose\"]" \
             | oc create -f -
  2. Look at output for core protocol pod
     $ oc get pods --watch
     $ oc logs -f {insert value of new pod name}
    
         ---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
         Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
         -----------------------------------------------------  -----------------------------------------------------  --------
              2.3              0           0       73     64.6       2.3              0           0       73     63.8         0
              4.3          3,337       1,667       71    100.0       4.3          2,529       1,261       75     96.6        42
              6.3         24,727      10,684       71    116.0       6.4         25,178      11,223      106    123.1        36
              8.3        113,170      44,177       64    116.7       8.4        116,495      45,658       66    123.9         2
             10.3        221,942      54,359       59    116.8      10.4        227,809      55,491       50    124.0         2
             12.3        327,983      52,994       56    116.8      12.4        332,016      52,077       44    122.3         2
             14.3        428,866      50,391       54    116.8      14.4        433,091      50,512       41    122.5         2
             16.3        540,823      55,923       62    117.1      16.4        547,056      56,926       49    122.6         2
             18.3        645,499      52,312       60    117.1      18.4        648,975      50,934       48    122.6         2
             20.3        761,704      58,073       58    117.1      20.4        768,000      59,453       55    122.7         2
             22.3        873,509      55,875       61    117.2      22.4        874,857      53,348       50    122.7         3
             24.3        914,014      20,242       33    117.2      24.5        923,768      22,728       27    122.7         4
             26.3      1,000,000      42,950       53    117.4      26.5      1,000,000      38,078       45      0.0         2
         --------------------------------------------------------------------------------
         Subject: activemq-artemis-jms //broker-amq-tcp:61616/activemq-artemis-jms (/tmp/quiver-xo6l0q4a)
         Count:                                          1,000,000 messages
         Body size:                                            100 bytes
         Credit window:                                      1,000 messages
         Duration:                                            23.0 s
         Sender rate:                                       43,414 messages/s
         Receiver rate:                                     43,507 messages/s
         End-to-end rate:                                   43,407 messages/s
         Latencies by percentile:
                   0%:        0 ms                90.00%:        4 ms
                  25%:        2 ms                99.00%:       20 ms
                  50%:        2 ms                99.90%:      249 ms
                 100%:      307 ms                99.99%:      290 ms
         --------------------------------------------------------------------------------
  3. Send 1,000,000 messages to the broker via the JMS client over AMQP protocol
     $ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
             DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
             DOCKER_CMD="[\"quiver\", \"amqp://broker-amq-amqp:5672/qpid-jms-over-ampq\", \"--impl\", \"qpid-jms\", \"--verbose\"]" \
             | oc create -f -
  4. Look at output for the AMQP client pod
     $ oc get pods --watch
     $ oc logs -f {insert value of new pod name}
    
         ---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
         Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
         -----------------------------------------------------  -----------------------------------------------------  --------
              2.2          2,663       1,330      109    103.1       2.2          1,984         991       94     94.6        89
              4.2         16,304       6,817      118    110.7       4.2         16,113       7,050      115    113.2        56
              6.2         45,720      14,693      102    119.7       6.2         45,809      14,767       97    115.4       118
              8.2        105,097      29,674       88    120.8       8.2         92,810      23,489       84    115.6       305
             10.2        159,018      26,920       71    120.8      10.2        146,602      26,883       76    115.6       411
             12.2        213,794      27,361       81    121.5      12.2        205,564      29,466       81    117.2       400
             14.2        271,138      28,643       77    121.6      14.2        256,132      25,271       76    117.2       461
             16.2        325,058      26,960       82    116.6      16.2        311,757      27,785       75    117.2       566
             18.2        383,747      29,315       78    116.6      18.2        362,224      25,208       72    117.2       678
             20.2        431,554      23,880       64    116.6      20.2        411,578      24,665       70    117.2       750
             22.2        492,444      30,430       75    116.6      22.2        468,113      28,225       77    117.2       786
             24.2        537,684      22,609       70    116.7      24.2        510,287      21,087       70    117.3     1,105
             26.2        595,639      28,963       77    116.7      26.2        567,833      28,759       75    117.4     1,049
             28.2        646,014      25,175       67    116.7      28.2        619,412      25,790       73    117.5     1,014
             30.2        701,401      27,680       73    116.9      30.2        669,677      25,095       72    117.5     1,167
             32.2        757,523      28,019       73    116.9      32.2        722,166      26,231       72    117.6     1,246
             34.2        803,007      22,731       70    117.1      34.2        768,587      23,210       66    117.6     1,360
             36.2        856,193      26,593       69    117.1      36.2        816,627      23,924       73    117.6     1,538
             38.2        911,459      27,605       73    117.1      38.2        870,633      26,990       77    117.6     1,633
             40.2        968,803      28,658       70    117.4      40.2        926,359      27,849       76    117.6     1,482
         ---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
         Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
         -----------------------------------------------------  -----------------------------------------------------  --------
             42.2      1,000,000      15,583       41      0.0      42.2        978,444      26,029       89    117.6     1,581
                -              -           -        -        -      44.2      1,000,000      10,773       57      0.0     1,608
         --------------------------------------------------------------------------------
         Subject: qpid-jms amqp://broker-amq-amqp:5672/qpid-jms-over-ampq (/tmp/quiver-a1xz3nbz)
         Count:                                          1,000,000 messages
         Body size:                                            100 bytes
         Credit window:                                      1,000 messages
         Duration:                                            42.4 s
         Sender rate:                                       24,696 messages/s
         Receiver rate:                                     23,616 messages/s
         End-to-end rate:                                   23,573 messages/s
         Latencies by percentile:
                   0%:        1 ms                90.00%:     1575 ms
                  25%:      534 ms                99.00%:     1741 ms
                  50%:      988 ms                99.90%:     1886 ms
                 100%:     1929 ms                99.99%:     1926 ms
         --------------------------------------------------------------------------------

Send messages to AMQ Interconnect

  1. Send 1,000,000 messages to the interconnect via the JMS client over AMQP protocol
    $ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
            DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
            DOCKER_CMD="[\"quiver\", \"amqp://amq-interconnect:5672/qpid-jms-over-ampq-via-interconnect\", \"--impl\", \"qpid-jms\", \"--verbose\"]" \
            | oc create -f -
  2. Look at output for AMQP client pod
    $ oc get pods --watch
    $ oc logs -f {insert value of new pod name}
    
        ---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
        Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
        -----------------------------------------------------  -----------------------------------------------------  --------
             2.2          1,529         764      131    102.6       2.2          1,465         731      120     84.5        21
             4.2         12,580       5,514      151    116.2       4.2         12,529       5,526      138    114.5        16
             6.2         42,121      14,756      125    119.5       6.2         41,815      14,606       90    111.3         6
             8.2         85,563      21,721       82    119.9       8.2         85,028      21,596       71    111.7         4
            10.2        125,149      19,773       75    120.2      10.2        124,858      19,905       62    112.0         5
            12.2        169,899      22,264       79    116.5      12.2        169,965      22,486       67    112.1         5
            14.3        212,816      21,437       69    116.7      14.2        212,442      21,217       60    112.1         5
            16.3        259,889      23,536       75    116.7      16.2        259,571      23,553       71    114.7         4
            18.3        308,674      24,332       86    119.1      18.2        308,723      24,539       69    114.9         4
            20.3        340,342      15,826       67    119.1      20.2        340,176      15,726       58    115.0         6
            22.3        386,804      23,208       76    119.3      22.2        386,597      23,199       67    115.0         4
            24.3        434,611      23,880       75    119.3      24.2        434,435      23,895       65    115.0         4
            26.3        472,270      18,783       63    119.4      26.2        471,754      18,650       53    115.0         6
            28.3        507,116      17,406       59    119.4      28.2        506,646      17,420       51    115.0         6
            30.3        541,841      17,345       57    119.4      30.2        541,639      17,488       56    115.1         6
            32.3        577,421      17,781       61    119.4      32.2        576,429      17,352       50    115.1         6
            34.3        612,512      17,528       61    119.4      34.2        612,231      17,892       53    115.1         6
            36.3        644,913      16,192       58    119.5      36.2        644,190      15,972       48    115.1         6
            38.3        678,782      16,901       58    119.5      38.2        678,577      17,176       52    115.1         7
            40.3        714,240      17,720       60    119.5      40.2        713,569      17,487       51    115.3         6
        ---------------------- Sender -----------------------  --------------------- Receiver ----------------------  --------
        Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Time [s]      Count [m]  Rate [m/s]  CPU [%]  RSS [M]  Lat [ms]
        -----------------------------------------------------  -----------------------------------------------------  --------
            42.3        750,187      17,965       62    119.5      42.2        748,967      17,672       51    115.3         6
            44.3        787,234      18,496       63    119.5      44.3        787,500      19,171       55    115.3         6
            46.3        820,124      16,420       58    119.5      46.3        819,155      15,820       48    115.3         6
            48.3        856,316      18,087       59    119.5      48.3        855,868      18,338       51    115.3         6
            50.3        897,520      20,551       68    119.5      50.3        897,232      20,641       57    115.3         5
            52.3        944,594      23,537       73    119.5      52.3        944,159      23,452       63    115.3         4
            54.3        990,567      22,986       72    119.5      54.3        990,176      22,997       64    115.3         4
            56.3      1,000,000       4,716       17      0.0      56.3      1,000,000       4,907       15      0.0         4
        --------------------------------------------------------------------------------
        Subject: qpid-jms amqp://amq-interconnect:5672/qpid-jms-over-ampq-via-interconnect (/tmp/quiver-0sttx8_z)
        Count:                                          1,000,000 messages
        Body size:                                            100 bytes
        Credit window:                                      1,000 messages
        Duration:                                            53.7 s
        Sender rate:                                       18,615 messages/s
        Receiver rate:                                     18,634 messages/s
        End-to-end rate:                                   18,614 messages/s
        Latencies by percentile:
                  0%:        0 ms                90.00%:       10 ms
                 25%:        3 ms                99.00%:       18 ms
                 50%:        5 ms                99.90%:       43 ms
                100%:       89 ms                99.99%:       72 ms
        --------------------------------------------------------------------------------

Cleanup

  1. Cleanup and delete all quiver pods
    $ oc delete pod -l app=quiver

Does your client use another language?

If your language of choice was not shown in the demo, the good news is that Quiver supports several implementations. Here are some examples:

$ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
        DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
        DOCKER_CMD="[\"quiver\", \"//broker-amq-tcp:61616/activemq-jms\", \"--impl\", \"activemq-jms\", \"--verbose\"]" \
        | oc create -f -

$ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
        DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
        DOCKER_CMD="[\"quiver\", \"//broker-amq-tcp:61616/qpid-messaging-cpp\", \"--impl\", \"qpid-messaging-cpp\", \"--verbose\"]" \
        | oc create -f -

$ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
        DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
        DOCKER_CMD="[\"quiver\", \"//broker-amq-tcp:61616/qpid-messaging-python\", \"--impl\", \"qpid-messaging-python\", \"--verbose\"]" \
        | oc create -f -

$ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
        DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
        DOCKER_CMD="[\"quiver\", \"//broker-amq-tcp:61616/qpid-proton-c\", \"--impl\", \"qpid-proton-c\", \"--verbose\"]" \
        | oc create -f -

$ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
        DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
        DOCKER_CMD="[\"quiver\", \"//broker-amq-tcp:61616/qpid-proton-cpp\", \"--impl\", \"qpid-proton-cpp\", \"--verbose\"]" \
        | oc create -f -

$ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
        DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
        DOCKER_CMD="[\"quiver\", \"//broker-amq-tcp:61616/qpid-proton-python\", \"--impl\", \"qpid-proton-python\", \"--verbose\"]" \
        | oc create -f -

$ oc process -f https://raw.githubusercontent.com/ssorj/quiver/0.2.0/packaging/openshift/openshift-pod-template.yml \
        DOCKER_IMAGE=$(oc get is quiver -o jsonpath='{.status.dockerImageRepository}'):latest \
        DOCKER_CMD="[\"quiver\", \"//broker-amq-tcp:61616/rhea\", \"--impl\", \"rhea\", \"--verbose\"]" \
        | oc create -f -

Food for thought

I hope this demo has shown how easy it is to use Quiver to interact with AMQ Broker and AMQ Interconnect on OCP.

It should have also raised questions around how Quiver could be integrated into your own systems. For example:

  • Could Quiver be part of your mvn integration tests?
  • Could Quiver be added as a stage within your CI/CD pipeline for smoke testing?
  • Could the results of the smoke test pass/fail the deployment?
  • Could Quiver be used as part of a heartbeat monitoring system to validate that the broker is alive?

These are several scenarios for which I see Quiver being highly useful, and you’ve probably thought of others.

Note

[1] Although Quiver is developed by Red Hat employees, it is not supported under a Red Hat subscription and is strictly an upstream project.

Last updated: November 14, 2023