Producing and consuming messages

Apache Kafka clients are available for most major programming languages to produce and consume messages. The example in this section uses the kcat CLI so that no custom coding is required to produce and consume messages.

Create a file named initial_songs and paste the following content into the file. These are two messages that will be produced and consumed from the topic:

1: {"id": 1, "name": "The Ecstasy of Gold", "author":"Ennio Morricone", "op":"ADD"}

2: {"id": 2, "name": "Still Loving you", "author":"Scorpions", "op":"ADD"}

Open a terminal and run the following command:

$ kcat -b $BOOTSTRAP_SERVER \
-X security.protocol=SASL_SSL \
-X sasl.mechanisms=PLAIN \
-X sasl.username="$CLIENT_ID" \
-X sasl.password="$CLIENT_SECRET" -t songs -C -K:

This instructs kcat to run in consumer mode, which will print incoming messages from the songs topic and their keys separated by a colon from their values.

Open a second terminal and run the following command to produce the songs into the songs topic. This command starts a producer (-P option) that writes to the songs topic (-t option), and reads the messages line by line (-l option) from the initial_songs file. Text before the colon (-K: option) on each line in the file is treated as the message’s key.

$ kcat -b $BOOTSTRAP_SERVER \

-X security.protocol=SASL_SSL \

-X sasl.mechanisms=PLAIN \

-X sasl.username="$CLIENT_ID" \

-X sasl.password="$CLIENT_SECRET" -t songs -P -K: -l initial_songs

The kcat consumer running in the first terminal should have consumed and printed the messages like so:

1: {"id": 1, "name": "The Ecstasy of Gold", "author":"Ennio Morricone", "op":"ADD"}

2: {"id": 2, "name": "Still Loving you", "author":"Scorpions", "op":"ADD"}

% Reached end of topic songs [0] at offset 2

Note that the offset of this consumer has been updated to 2, as the two first records have been consumed from the topic.