Tuesday, 19 June 2018

Step by Step how to upgrade Python 2.6.x to 2.7.x with Kafka module.

Step1:-

[root@sankar-devops]# python --version
Python 2.6.6

[root@sankar-devops]# python 
Python 2.6.6 (r266:84292, Aug 18 2016, 08:36:59) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.


Step2:
[root@sankar-devops]#cd /opt

[root@sankar-devops opt]#wget http://www.python.org/ftp/python/2.7.8/Python-2.7.8.tar.xz  

[root@sankar-devops opt]#xz -d Python-2.7.8.tar.xz  

[root@sankar-devops opt]#tar -xvf Python-2.7.8.tar 

Step3: [root@sankar-devops opt]# cd Python-2.7.8
[root@sankar-devops Python-2.7.8]# ./configure --prefix=/usr/local
....
....
....
checking for %zd printf() format support... yes
checking for socklen_t... yes
checking for build directories... done
configure: creating ./config.status
config.status: creating Makefile.pre
config.status: creating Modules/Setup.config
config.status: creating Misc/python.pc
config.status: creating Modules/ld_so_aix
config.status: creating pyconfig.h
creating Modules/Setup
creating Modules/Setup.local
creating Makefile


[root@sankar-devops Python-2.7.8]# make

[root@sankar-devops Python-2.7.8]# make altinstall
running install_scripts
copying build/scripts-2.7/pydoc -> /usr/local/bin
copying build/scripts-2.7/smtpd.py -> /usr/local/bin
copying build/scripts-2.7/idle -> /usr/local/bin
copying build/scripts-2.7/2to3 -> /usr/local/bin
changing mode of /usr/local/bin/pydoc to 755
changing mode of /usr/local/bin/smtpd.py to 755
changing mode of /usr/local/bin/idle to 755
changing mode of /usr/local/bin/2to3 to 755
running install_egg_info
Writing /usr/local/lib/python2.7/lib-dynload/Python-2.7.8-py2.7.egg-info
rm /usr/local/lib/python2.7/lib-dynload/_sysconfigdata.py*

Step4:
Put the below line in ~/.bashrc file:

[root@sankar-devops]#vim ~/.bashrc
....
....
alias python=/usr/local/bin/python2.7

:wq!

Now execute the command:

[root@sankar-devops]#source ~/.bashrc


Step5:
Kafka Module installation on Python 2.7.x

[root@sankar-devops]#cd /usr/local/lib/python2.7/site-packages


[root@sankar-devops]#cd /usr/lib/python2.6/

[root@sankar-devops]#cp -r kafka /usr/local/lib/python2.7/site-packages/

[root@sankar-devops]#cp -r kafka_python-1.4.4.dev-py2.6.egg-info/  /usr/local/lib/python2.7/site-packages/

[root@sankar-devops]#cp -r pip*  /usr/local/lib/python2.7/site-packages/

  
Step6: Verification
[root@sankar-devops]# python
Python 2.7.8 (default, Jun 18 2018, 11:10:21) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> help()

Welcome to Python 2.7!  This is the online help utility.


help> modules

Please wait a moment while I gather a list of all available modules...

BaseHTTPServer      ast                 imp                 shelve
Bastion             asynchat            importlib           shlex
Cookie              base64              json                smtplib
DLFCN               bdb                 kafka               sndhdr
Dialog              binascii            keyword             socket
DocXMLRPCServer     binhex              lib2to3             spwd
FileDialog          bisect              linecache           sqlite3


help> modules kafka

Here is a list of matching modules.  Enter any module name to get more help.

kafka 
kafka.client 
kafka.client_async 
kafka.cluster 
kafka.codec 
kafka.common 
kafka.conn 
kafka.consumer 
kafka.consumer.base 
kafka.consumer.fetcher 

Thursday, 31 May 2018

PART2- Single Node-Single & Multi Broker Configuration

PART1-Kafka

Single Node-Single Broker Configuration

Step:1
Creating a Kafka Topic 

syntax:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic-name

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Hello-Sankar-Kafka
Created topic "Hello-Sankar-Kafka".

Step:2
Once the topic has been created, you can get the notification in Kafka broker terminal window and the log for the created topic specified in “/tmp/kafka-logs/“ in the config/server.properties file.

[root@sankar-devops kafka_2.12-1.1.0]# cd /tmp/kafka-logs/
[root@sankar-devops kafka-logs]# ls -ltrh
total 16K
-rw-r--r--. 1 root root   0 May 31 03:39 cleaner-offset-checkpoint
-rw-r--r--. 1 root root  54 May 31 03:39 meta.properties
drwxr-xr-x. 2 root root 141 May 31 05:19 Hello-Sankar-Kafka-0
-rw-r--r--. 1 root root  27 May 31 05:21 recovery-point-offset-checkpoint
-rw-r--r--. 1 root root   4 May 31 05:21 log-start-offset-checkpoint
-rw-r--r--. 1 root root  27 May 31 05:22 replication-offset-checkpoint
[root@sankar-devops kafka-logs]#

Step:3
List of Topics
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Hello-Sankar-Kafka
[root@sankar-devops kafka_2.12-1.1.0]#

Step:4
Start Producer to Send Messages
[root@sankar-devops kafka_2.12-1.1.0]# netstat -antlup |grep 9092
tcp6       0      0 :::9092                 :::*                    LISTEN      11726/java       
tcp6       0      0 10.219.39.148:9092      10.219.39.148:44332     ESTABLISHED 11726/java       
tcp6       0      0 10.219.39.148:44332     10.219.39.148:9092      ESTABLISHED 11726/java       
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello-Sankar-Kafka
>hello Sankar
>my first message
>my second message
>


Start Consumer to Receive Messages

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Hello-Sankar-Kafka --from-beginning
hello Sankar
my first message
my second message


Step:5
Single Node-Multiple Brokers Configuration

[root@sankar-devops config]# vi server01.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
# The port the socket server listens on
port=9093
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-1


[root@sankar-devops config]# vi server02.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2
# The port the socket server listens on
port=9094
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-2


[root@sankar-devops ~]# bin/kafka-server-start.sh config/server01.properties &
[1] 14077

[root@sankar-devops ~]# bin/kafka-server-start.sh config/server02.properties &
[1] 14078

Now we have three different brokers running on the machine.

FYI,
[root@sankar-devops kafka_2.12-1.1.0]# jps
14081 Kafka
13752 Kafka
14681 Jps
12253 QuorumPeerMain
11726 Kafka
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# netstat -ntlup|grep java   
tcp6       0      0 :::9092                 :::*                    LISTEN      11726/java       
tcp6       0      0 :::9093                 :::*                    LISTEN      13752/java       
tcp6       0      0 :::2181                 :::*                    LISTEN      12253/java       
tcp6       0      0 :::9094                 :::*                    LISTEN      14081/java

Step:6
Creating a Topic:-
We have three different brokers running, Let us assign the replication factor value as three for this topic.

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic Sankar-Multibrokerapplication
Created topic "Sankar-Multibrokerapplication".

Step:7
To check which broker is listening on the current created topic

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication
Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:              //summary of all the partitions
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2    //each node will be the leader for a randomly selected portion of the partitions.

FYI,
for this case, we see that our first broker ( broker.id 0) is the leader.
Then Replicas:0,1,2 means that all the brokers replicate the topic
finally Isr is the set of in-sync replicas, this is the subset of replicas that are currently alive and caught up by the leader.

Step:8
Start Producer to Send Messages, Let's publish a few messages to our new topic:)

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Sankar-Multibrokerapplication
>this is signle node and multip broker demo

>this is second message
>this is sai
>this is lohi
>this is moni
>^C[root@sankar-devops kafka_2.12-1.1.0]#

Step:9
Start Consumer to Receive Messages, Now let's consume these messages:

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Sankar-Multibrokerapplication --from-beginning
this is signle node and multip broker demo

this is second message
this is sai
this is lohi
this is moni


Step:10
Now let's test out fault-tolerance. Broker 0 was acting as the leader so let's kill it:

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication
Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:           
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2 


[root@sankar-devops kafka_2.12-1.1.0]# jps
14081 Kafka
16338 Jps
13752 Kafka
12253 QuorumPeerMain
11726 Kafka
[root@sankar-devops kafka_2.12-1.1.0]# kill -9 11726

Leadership has switched to one of the slaves and node 1 is no longer in the in-sync replica set:

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication

Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 1       Replicas: 0,1,2 Isr: 1,2

errors:
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Sankar-Multibrokerapplication --from-beginning
[2018-05-31 06:44:31,809] WARN [Consumer clientId=consumer-1, groupId=console-consumer-89327] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)


Now i am up the broker-id.0(server.properties), Leader was not change.
FYI,
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication

Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 1       Replicas: 0,1,2 Isr: 1,2,0

Step:11
Basic Topic Operations like Modifying a Topic.

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Hello-Sankar-Kafka
Sankar-Multibrokerapplication
__consumer_offsets
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Hello-Sankar-Kafka
Topic:Hello-Sankar-Kafka        PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: Hello-Sankar-Kafka       Partition: 0    Leader: 0       Replicas: 0     Isr: 0
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic Hello-Sankar-Kafka --partitions 2
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Hello-Sankar-Kafka
Topic:Hello-Sankar-Kafka        PartitionCount:2        ReplicationFactor:1     Configs:
        Topic: Hello-Sankar-Kafka       Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: Hello-Sankar-Kafka       Partition: 1    Leader: 1       Replicas: 1     Isr: 1
[root@sankar-devops kafka_2.12-1.1.0]#

Step:12
Deleting a Topic
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Hello-Sankar-Kafka
Sankar-Multibrokerapplication
__consumer_offsets
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic Hello-Sankar-Kafka
Topic Hello-Sankar-Kafka is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Sankar-Multibrokerapplication
__consumer_offsets