Single node setup
apt install default-jre
wget https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz
tar -xzf kafka_2.12–2.2.0.tgz
cd kafka_2.12–2.2.0
#start zk
bin/zookeeper-server-start.sh config/zookeeper.properties
#start kafka
bin/kafka-server-start.sh config/server.properties
#create a topic
bin/kafka-topics.sh — create — zookeeper localhost:2181 — replication-factor 1 — partitions 1 — topic first_topic
#list topic
bin/kafka-topics.sh — list — zookeeper localhost:2181
#produce a message
echo “hello” | bin/kafka-console-producer.sh — broker-list localhost:9092 — topic first_topic > /dev/null
#consume the message
bin/kafka-console-consumer.sh — bootstrap-server localhost:9092 — topic first_topic — from-beginning
Now Cluster, what are we going to setup
2 VM : 192.168.11.137(server1), 192.168.11.138(server2)
2 kafka installation (server1, server2)
1 zookeeper (server1)
Download the same tar file
tar -xzf kafka_2.12–2.2.0.tgz
cd kafka_2.12–2.2.0
modify setting
sudo nano /config/server.properties
change below lines(broker.id and listener ip is unique)
Broker.id=2
Zookeer.connect = 192.168.11.137:2181
listeners=PLAINTEXT://192.168.11.138:9092
run zookeeper on first server (192.168.11.137, will make zookeeper cluster next time).
bin/zookeeper-server-start.sh config/zookeeper.properties
start kafka server2(Connect to remote zookeeper)
bin/kafka-server-start.sh config/server.properties
Create second topic for cluster
sudo bin/kafka-topics.sh — create — zookeeper 192.168.11.137:2181 — replication-factor 2 — partitions 1 — topic second_topic
Run producer (connect to server 1)
sudo ./bin/kafka-console-producer.sh — broker-list 192.168.11.137:9092 — topic second_topic
produce some message to test
Now start 2 consumer
Consumer 1
Consumer 2
You may try create producer connect to server 2 (192.168.11.138).
