PART2- Single Node-Single & Multi Broker Configuration

PART1-Kafka

Single Node-Single Broker Configuration

Step:1
Creating a Kafka Topic 

syntax:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic-name

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Hello-Sankar-Kafka
Created topic "Hello-Sankar-Kafka".

Step:2
Once the topic has been created, you can get the notification in Kafka broker terminal window and the log for the created topic specified in “/tmp/kafka-logs/“ in the config/server.properties file.

[root@sankar-devops kafka_2.12-1.1.0]# cd /tmp/kafka-logs/
[root@sankar-devops kafka-logs]# ls -ltrh
total 16K
-rw-r--r--. 1 root root   0 May 31 03:39 cleaner-offset-checkpoint
-rw-r--r--. 1 root root  54 May 31 03:39 meta.properties
drwxr-xr-x. 2 root root 141 May 31 05:19 Hello-Sankar-Kafka-0
-rw-r--r--. 1 root root  27 May 31 05:21 recovery-point-offset-checkpoint
-rw-r--r--. 1 root root   4 May 31 05:21 log-start-offset-checkpoint
-rw-r--r--. 1 root root  27 May 31 05:22 replication-offset-checkpoint
[root@sankar-devops kafka-logs]#

Step:3
List of Topics
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Hello-Sankar-Kafka
[root@sankar-devops kafka_2.12-1.1.0]#

Step:4
Start Producer to Send Messages
[root@sankar-devops kafka_2.12-1.1.0]# netstat -antlup |grep 9092
tcp6       0      0 :::9092                 :::*                    LISTEN      11726/java       
tcp6       0      0 10.219.39.148:9092      10.219.39.148:44332     ESTABLISHED 11726/java       
tcp6       0      0 10.219.39.148:44332     10.219.39.148:9092      ESTABLISHED 11726/java       
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello-Sankar-Kafka
>hello Sankar
>my first message
>my second message
>


Start Consumer to Receive Messages

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Hello-Sankar-Kafka --from-beginning
hello Sankar
my first message
my second message


Step:5
Single Node-Multiple Brokers Configuration

[root@sankar-devops config]# vi server01.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
# The port the socket server listens on
port=9093
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-1


[root@sankar-devops config]# vi server02.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2
# The port the socket server listens on
port=9094
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-2


[root@sankar-devops ~]# bin/kafka-server-start.sh config/server01.properties &
[1] 14077

[root@sankar-devops ~]# bin/kafka-server-start.sh config/server02.properties &
[1] 14078

Now we have three different brokers running on the machine.

FYI,
[root@sankar-devops kafka_2.12-1.1.0]# jps
14081 Kafka
13752 Kafka
14681 Jps
12253 QuorumPeerMain
11726 Kafka
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# netstat -ntlup|grep java   
tcp6       0      0 :::9092                 :::*                    LISTEN      11726/java       
tcp6       0      0 :::9093                 :::*                    LISTEN      13752/java       
tcp6       0      0 :::2181                 :::*                    LISTEN      12253/java       
tcp6       0      0 :::9094                 :::*                    LISTEN      14081/java

Step:6
Creating a Topic:-
We have three different brokers running, Let us assign the replication factor value as three for this topic.

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic Sankar-Multibrokerapplication
Created topic "Sankar-Multibrokerapplication".

Step:7
To check which broker is listening on the current created topic

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication
Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:              //summary of all the partitions
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2    //each node will be the leader for a randomly selected portion of the partitions.

FYI,
for this case, we see that our first broker ( broker.id 0) is the leader.
Then Replicas:0,1,2 means that all the brokers replicate the topic
finally Isr is the set of in-sync replicas, this is the subset of replicas that are currently alive and caught up by the leader.

Step:8
Start Producer to Send Messages, Let's publish a few messages to our new topic:)

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Sankar-Multibrokerapplication
>this is signle node and multip broker demo

>this is second message
>this is sai
>this is lohi
>this is moni
>^C[root@sankar-devops kafka_2.12-1.1.0]#

Step:9
Start Consumer to Receive Messages, Now let's consume these messages:

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Sankar-Multibrokerapplication --from-beginning
this is signle node and multip broker demo

this is second message
this is sai
this is lohi
this is moni


Step:10
Now let's test out fault-tolerance. Broker 0 was acting as the leader so let's kill it:

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication
Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:           
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2 


[root@sankar-devops kafka_2.12-1.1.0]# jps
14081 Kafka
16338 Jps
13752 Kafka
12253 QuorumPeerMain
11726 Kafka
[root@sankar-devops kafka_2.12-1.1.0]# kill -9 11726

Leadership has switched to one of the slaves and node 1 is no longer in the in-sync replica set:

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication

Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 1       Replicas: 0,1,2 Isr: 1,2

errors:
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Sankar-Multibrokerapplication --from-beginning
[2018-05-31 06:44:31,809] WARN [Consumer clientId=consumer-1, groupId=console-consumer-89327] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)


Now i am up the broker-id.0(server.properties), Leader was not change.
FYI,
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication

Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 1       Replicas: 0,1,2 Isr: 1,2,0

Step:11
Basic Topic Operations like Modifying a Topic.

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Hello-Sankar-Kafka
Sankar-Multibrokerapplication
__consumer_offsets
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Hello-Sankar-Kafka
Topic:Hello-Sankar-Kafka        PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: Hello-Sankar-Kafka       Partition: 0    Leader: 0       Replicas: 0     Isr: 0
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic Hello-Sankar-Kafka --partitions 2
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Hello-Sankar-Kafka
Topic:Hello-Sankar-Kafka        PartitionCount:2        ReplicationFactor:1     Configs:
        Topic: Hello-Sankar-Kafka       Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: Hello-Sankar-Kafka       Partition: 1    Leader: 1       Replicas: 1     Isr: 1
[root@sankar-devops kafka_2.12-1.1.0]#

Step:12
Deleting a Topic
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Hello-Sankar-Kafka
Sankar-Multibrokerapplication
__consumer_offsets
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic Hello-Sankar-Kafka
Topic Hello-Sankar-Kafka is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Sankar-Multibrokerapplication
__consumer_offsets