Kafdrop: I can see you, Protobuf

 

Kafdrop is a web UI built on top of Spring Boot. It's free and has a good dark theme. (Well, the dark theme is the only theme it provides, actually.) Speaking of kafka web UI, there is another web UI like the Kafka UI - which also has the capability of browsing message, but at the time I write this blog post Kafdrop is the only one that supports viewing protobuf message.

Since I've been dealing a lot with kafka protobuf messages lately, I needed to know how to use Kafdrop with protobuf support. Because you know, it's such a pain in the eye viewing protobuf using kafkacat and protoc:

kafkacat -b kafka:9092 -C -t topic.name -o beginning -e | protoc --decode_raw

Luckily after few days of fooling around, I found that making kafdrop to work with protobuf is not a hard thing to do.

The Kafdrop Readme says that I need to put the protobuf descriptor file (.desc) to a directory, and pass the directory path as a parameter to Kafdrop. Creating a descriptor file from a .proto file is straight forward. I only need to do this:

protoc -o outfile.desc -I import-path-1:import-path-2 --include_imports \
file-1.proto file-2.proto

--include_imports option only necessary when your proto imports another file.

In case you are not familiar with protoc, in Ubuntu protoc can be installed using:

sudo apt update && sudo apt install -y protobuf-compiler

Once I have the .desc file in the right location, there's only one more thing to do. Pass the directory as a protobufdesc.directory parameter to Kafdrop. When I want to peek a message in a remote server, I prefer using Kafdrop jar like the following, but please beware that running Kafdrop jar requires Java 11:

java --add-opens=java.base/sun.nio.ch=ALL-UNNAMED -jar $KAFDROP_JAR \
--kafka.brokerConnect=remote.kafka:9092 --message.format=PROTOBUF \
--protobufdesc.directory=$PROTOBUFDESC_DIR

 Or, with docker-compose.yaml from Kafdrop repo :

services:
  kafdrop:
    ...
    environment:
      KAFKA_BROKERCONNECT: remote.kafka:9092
      CMD_ARGS: "--message.format=PROTOBUF --protobufdesc.directory=/var/protobuf_desc"
    ...
    volumes:
      - ~/protobuf/desc:/var/protobuf_desc

Then, when I navigate my browser to localhost:9000, click the specified topic's link, click the "View Messages" button, and pick PROTOBUF from the Message format dropdown list, pick the .desc file, and last but not least fill in the message name and click the button next to it. I can see the protobuf messages beautifully formatted.

Try this in your machine, clone my git repo here.