Saturday, February 3, 2018

Important Kafka Commands

Important Kafka Commands

MindTelligent has developed an open source Kafka Administration Framework. This framework monitors the health of Kafka And Zookeeper nodes. It also ships a bunch of important Kafka commands.


These commands can be executed from $KAFKA_HOME/bin directory


Command to Set the Kafka Topic retention Period for 10 days:


./kafka-topics.sh --zookeeper localhost:2181 --alter --topic mindtelligent_topic  --config retention.ms=864000000

Command to Set Kafka Kafka partitions to 6 :


./kafka-topics.sh --zookeeper localhost:2181 --alter --topic mindtelligent_topic --partitions 6

Command to View offsets for the Kafka Consumer Group and instances for a consumer group:


./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group mindtelligent_topic_group


Command to list all Kafka Consumer Groups across all topics:

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list


Command to set the Kafka Offset to earliest:


./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group mindtelligent_topic_group --reset-offsets --to-earliest --topic mindtelligent_topic --execute


Command to set the Kafka Offset to Latest:


./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group mindtelligent_topic_group --reset-offsets --to-latest --topic mindtelligent_topic --execute

List Kafka Topic:


./kafka-topics.sh --zookeeper localhost:2181 --list



Describe a Kafka Topic:


./kafka-topics.sh --zookeeper localhost:2181 --describe --topic mindtelligent_topic

Purge a Kafka Topic:


./kafka-topics.sh --zookeeper localhost:2181 --alter --topic mindtelligent_topic--config retention.ms=1000

Delete a Kafka Topic: 

./kafka-topics.sh --zookeeper localhost:2181 --delete --topic mindtelligent_topic

Get Number of Messages in a Kafka Toipc:


./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mindtelligent_topic--time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'


Get the earliest offset still in a topic:

./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mindtelligent_topic  --time -2

Get the latest offset still in a topic:

./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mindtelligent_topic --time -1




OCI Knowledge Series: OCI Infrastructure components

  Oracle Cloud Infrastructure (OCI) provides a comprehensive set of infrastructure services that enable you to build and run a wide range of...