New Relic Installation & Configuration Steps

What is New Relic:

• Lew Cirne founded New Relic in 2008.
• New Relic's technology, delivered in a software as a service (SaaS) model, was announced in 2013.
• It monitors Web and mobile applications in real-time with support for custom-built plugins to collect arbitrary data.

New Relic Free Trail Version: Simple PHP Application Monitoring 

New Relic Installation & Configuration Steps:

Click on Free Trail and fill the below

Once update the info then next login into below Page.

Then Click on Add a User. And add new user with admin privileges

[root@sankar-paas-1 ~]# firewall-cmd --state

not running

[root@sankar-paas ~]# httpd -v

Server version: Apache/2.4.6 (CentOS)

Server built:   Nov  5 2018 01:47:09

[root@sankar-paas ~]# systemctl start httpd

[root@sankar-paas ~]# systemctl status httpd

● httpd.service - The Apache HTTP Server

   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)

   Active: active (running) since Mon 2018-12-24 19:55:12 UTC; 4s ago

     Docs: man:httpd(8)

           man:apachectl(8)

 Main PID: 1560 (httpd)

   Status: "Processing requests..."

   CGroup: /system.slice/httpd.service

           ├─1560 /usr/sbin/httpd -DFOREGROUND

           ├─1561 /usr/sbin/httpd -DFOREGROUND

           ├─1562 /usr/sbin/httpd -DFOREGROUND

           ├─1563 /usr/sbin/httpd -DFOREGROUND

           ├─1564 /usr/sbin/httpd -DFOREGROUND

           └─1565 /usr/sbin/httpd -DFOREGROUND

Dec 24 19:55:12 sankar-paas  systemd[1]: Starting The Apache HTTP Server...

Dec 24 19:55:12 sankar-paas  systemd[1]: Started The Apache HTTP Server.

[root@sankar-paas  ~]#

[root@sankar-paas ~]# yum install php php-mysql php-pdo php-gd php-mbstring

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.scalabledns.com

 * epel: d2lzkl7pfhq30w.cloudfront.net

 * extras: mirror.fileplanet.com

 * updates: mirrors.oit.uci.edu

Resolving Dependencies

--> Running transaction check

---> Package php.x86_64 0:5.4.16-46.el7 will be installed

--> Processing Dependency: php-common(x86-64) = 5.4.16-46.el7 for package: php-5.4.16-46.el7.x86_64

--> Processing Dependency: php-cli(x86-64) = 5.4.16-46.el7 for package: php-5.4.16-46.el7.x86_64

---> Package php-gd.x86_64 0:5.4.16-46.el7 will be installe[root@greyl-1 ~]# echo "<?php phpinfo(); ?>" > /var/www/html/info.php

….

….

[root@sankar-paas ~]# echo "<?php phpinfo(); ?>" > /var/www/html/info.php

[root@sankar-paas ~]# systemctl restart httpd

Install the PHP agent

[root@sankar-paas conf.d]# rpm -Uvh http://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpm

Retrieving http://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpm

warning: /var/tmp/rpm-tmp.CK11kK: Header V3 DSA/SHA1 Signature, key ID 548c16bf: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:newrelic-repo-5-3                ################################# [100%]

[root@sankar-paas conf.d]# yum install newrelic-php5

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.fileplanet.com

 * epel: d2lzkl7pfhq30w.cloudfront.net

 * extras: mirror.fileplanet.com

 * updates: repos.lax.quadranet.com

newrelic                                                                                                                                     | 2.5 kB  00:00:00    

newrelic/x86_64/primary_db                                                                                                                   |  55 kB  00:00:00    

Resolving Dependencies

--> Running transaction check

---> Package newrelic-php5.x86_64 0:8.5.0.235-1 will be installed

….

…

[root@sankar-paas conf.d]# newrelic-install install

New Relic PHP Agent Installation (interactive mode)

===================================================

   Enter New Relic license key (or leave blank): ******************************3a17

Below is a list of the directories in which we found a copy of PHP.

Please select the directory or directories for which you wish to install

New Relic. You can select either a single directory or

multiple directories by separating each choice with either a space or a

comma. To select all of the directories shown, please enter the special

keyword 'all' (without the quotes).

   1)   /bin

   2)   /usr/bin

   0)   Exit

   Selection (1-2, 0 to exit or all): 2

   Found a valid PHP in : /usr/bin

   PHP Version : 5.4.16

   Module API version : 20100525

   Module directory : /usr/lib64/php/modules

   Zend Thread Safety : no

   CLI ini directory : /etc/php.d

   Install Status : OK

New Relic is now installed on your system. Congratulations!

1. Set newrelic.appname in your newrelic.ini file.

2. Restart your web server. This will fix most reporting issues and

   load the agent's new features and bug fixes.

If you have questions or comments, go to http://support.newrelic.com.

http://ip:80

Load Test :

[root@sankar-paas  ~]# curl -s "http://**.**.**.**/info.php?[1-1000]"

su: warning: cannot change directory to /home/user: No such file or directory(/home/pmta1)

su: warning: cannot change directory to /home/user(pmta1): No such file or directory(/home/pmta1)

My Case: user is : pmta1

[root@sankar-sysops]# su - pmta1
su: warning: cannot change directory to /home/pmta1: No such file or directory

-bash-4.1$ pwd
/root

-bash-4.1$ exit
logout

[root@sankar-sysops]#  cat /etc/passwd |grep pmta1
pmta1:x:701:701::/home/pmta:/bin/bash

[root@sankar-sysops]# ls -l /home/pmta1  (no directory existed)

I've add below entry to /etc/pam.d/system-auth

session optional /lib/security/$ISA/pam_mkhomedir.so


[root@sankar-sysops]#  vim /etc/pam.d/system-auth
...
...
...
session optional /lib/security/$ISA/pam_mkhomedir.so

:wq!

[root@sankar-sysops]# su - pmta1
Creating directory '/home/pmta1'

[pmta1@sankar-sysops]$

Step by Step how to upgrade Python 2.6.x to 2.7.x with Kafka module.

Step1:-

[root@sankar-devops]# python --version
Python 2.6.6

[root@sankar-devops]# python 
Python 2.6.6 (r266:84292, Aug 18 2016, 08:36:59) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.


Step2:
[root@sankar-devops]#cd /opt

[root@sankar-devops opt]#wget http://www.python.org/ftp/python/2.7.8/Python-2.7.8.tar.xz  

[root@sankar-devops opt]#xz -d Python-2.7.8.tar.xz  

[root@sankar-devops opt]#tar -xvf Python-2.7.8.tar 

Step3: [root@sankar-devops opt]# cd Python-2.7.8
[root@sankar-devops Python-2.7.8]# ./configure --prefix=/usr/local
....
....
....
checking for %zd printf() format support... yes
checking for socklen_t... yes
checking for build directories... done
configure: creating ./config.status
config.status: creating Makefile.pre
config.status: creating Modules/Setup.config
config.status: creating Misc/python.pc
config.status: creating Modules/ld_so_aix
config.status: creating pyconfig.h
creating Modules/Setup
creating Modules/Setup.local
creating Makefile


[root@sankar-devops Python-2.7.8]# make

[root@sankar-devops Python-2.7.8]# make altinstall
running install_scripts
copying build/scripts-2.7/pydoc -> /usr/local/bin
copying build/scripts-2.7/smtpd.py -> /usr/local/bin
copying build/scripts-2.7/idle -> /usr/local/bin
copying build/scripts-2.7/2to3 -> /usr/local/bin
changing mode of /usr/local/bin/pydoc to 755
changing mode of /usr/local/bin/smtpd.py to 755
changing mode of /usr/local/bin/idle to 755
changing mode of /usr/local/bin/2to3 to 755
running install_egg_info
Writing /usr/local/lib/python2.7/lib-dynload/Python-2.7.8-py2.7.egg-info
rm /usr/local/lib/python2.7/lib-dynload/_sysconfigdata.py*

Step4:
Put the below line in ~/.bashrc file:

[root@sankar-devops]#vim ~/.bashrc
....
....
alias python=/usr/local/bin/python2.7

:wq!

Now execute the command:

[root@sankar-devops]#source ~/.bashrc


Step5: Kafka Module installation on Python 2.7.x

[root@sankar-devops]#cd /usr/local/lib/python2.7/site-packages


[root@sankar-devops]#cd /usr/lib/python2.6/

[root@sankar-devops]#cp -r kafka /usr/local/lib/python2.7/site-packages/

[root@sankar-devops]#cp -r kafka_python-1.4.4.dev-py2.6.egg-info/  /usr/local/lib/python2.7/site-packages/

[root@sankar-devops]#cp -r pip*  /usr/local/lib/python2.7/site-packages/

  
Step6: Verification
[root@sankar-devops]# python
Python 2.7.8 (default, Jun 18 2018, 11:10:21) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> help()

Welcome to Python 2.7!  This is the online help utility.


help> modules

Please wait a moment while I gather a list of all available modules...

BaseHTTPServer      ast                 imp                 shelve
Bastion             asynchat            importlib           shlex
Cookie              base64              json                smtplib
DLFCN               bdb                 kafka               sndhdr
Dialog              binascii            keyword             socket
DocXMLRPCServer     binhex              lib2to3             spwd
FileDialog          bisect              linecache           sqlite3


help> modules kafka

Here is a list of matching modules.  Enter any module name to get more help.

kafka 
kafka.client 
kafka.client_async 
kafka.cluster 
kafka.codec 
kafka.common 
kafka.conn 
kafka.consumer 
kafka.consumer.base 
kafka.consumer.fetcher 

PART2- Single Node-Single & Multi Broker Configuration

PART1-Kafka

Single Node-Single Broker Configuration

Step:1
Creating a Kafka Topic 

syntax:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic-name

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Hello-Sankar-Kafka
Created topic "Hello-Sankar-Kafka".

Step:2
Once the topic has been created, you can get the notification in Kafka broker terminal window and the log for the created topic specified in “/tmp/kafka-logs/“ in the config/server.properties file.

[root@sankar-devops kafka_2.12-1.1.0]# cd /tmp/kafka-logs/
[root@sankar-devops kafka-logs]# ls -ltrh
total 16K
-rw-r--r--. 1 root root   0 May 31 03:39 cleaner-offset-checkpoint
-rw-r--r--. 1 root root  54 May 31 03:39 meta.properties
drwxr-xr-x. 2 root root 141 May 31 05:19 Hello-Sankar-Kafka-0
-rw-r--r--. 1 root root  27 May 31 05:21 recovery-point-offset-checkpoint
-rw-r--r--. 1 root root   4 May 31 05:21 log-start-offset-checkpoint
-rw-r--r--. 1 root root  27 May 31 05:22 replication-offset-checkpoint
[root@sankar-devops kafka-logs]#

Step:3
List of Topics
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Hello-Sankar-Kafka
[root@sankar-devops kafka_2.12-1.1.0]#

Step:4
Start Producer to Send Messages
[root@sankar-devops kafka_2.12-1.1.0]# netstat -antlup |grep 9092
tcp6       0      0 :::9092                 :::*                    LISTEN      11726/java       
tcp6       0      0 10.219.39.148:9092      10.219.39.148:44332     ESTABLISHED 11726/java       
tcp6       0      0 10.219.39.148:44332     10.219.39.148:9092      ESTABLISHED 11726/java       
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello-Sankar-Kafka
>hello Sankar
>my first message
>my second message
>


Start Consumer to Receive Messages

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Hello-Sankar-Kafka --from-beginning
hello Sankar
my first message
my second message


Step:5
Single Node-Multiple Brokers Configuration

[root@sankar-devops config]# vi server01.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
# The port the socket server listens on
port=9093
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-1


[root@sankar-devops config]# vi server02.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2
# The port the socket server listens on
port=9094
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-2


[root@sankar-devops ~]# bin/kafka-server-start.sh config/server01.properties &
[1] 14077

[root@sankar-devops ~]# bin/kafka-server-start.sh config/server02.properties &
[1] 14078

Now we have three different brokers running on the machine.

FYI,
[root@sankar-devops kafka_2.12-1.1.0]# jps
14081 Kafka
13752 Kafka
14681 Jps
12253 QuorumPeerMain
11726 Kafka
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# netstat -ntlup|grep java   
tcp6       0      0 :::9092                 :::*                    LISTEN      11726/java       
tcp6       0      0 :::9093                 :::*                    LISTEN      13752/java       
tcp6       0      0 :::2181                 :::*                    LISTEN      12253/java       
tcp6       0      0 :::9094                 :::*                    LISTEN      14081/java

Step:6
Creating a Topic:-
We have three different brokers running, Let us assign the replication factor value as three for this topic.

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic Sankar-Multibrokerapplication
Created topic "Sankar-Multibrokerapplication".

Step:7
To check which broker is listening on the current created topic

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication
Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:              //summary of all the partitions
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2    //each node will be the leader for a randomly selected portion of the partitions.

FYI,
for this case, we see that our first broker ( broker.id 0) is the leader.
Then Replicas:0,1,2 means that all the brokers replicate the topic
finally Isr is the set of in-sync replicas, this is the subset of replicas that are currently alive and caught up by the leader.

Step:8
Start Producer to Send Messages, Let's publish a few messages to our new topic:)

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Sankar-Multibrokerapplication
>this is signle node and multip broker demo

>this is second message
>this is sai
>this is lohi
>this is moni
>^C[root@sankar-devops kafka_2.12-1.1.0]#

Step:9
Start Consumer to Receive Messages, Now let's consume these messages:

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Sankar-Multibrokerapplication --from-beginning
this is signle node and multip broker demo

this is second message
this is sai
this is lohi
this is moni


Step:10
Now let's test out fault-tolerance. Broker 0 was acting as the leader so let's kill it:

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication
Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:           
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 0       Replicas: 0,1,2 Isr: 0,1,2 


[root@sankar-devops kafka_2.12-1.1.0]# jps
14081 Kafka
16338 Jps
13752 Kafka
12253 QuorumPeerMain
11726 Kafka
[root@sankar-devops kafka_2.12-1.1.0]# kill -9 11726

Leadership has switched to one of the slaves and node 1 is no longer in the in-sync replica set:

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication

Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 1       Replicas: 0,1,2 Isr: 1,2

errors:
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Sankar-Multibrokerapplication --from-beginning
[2018-05-31 06:44:31,809] WARN [Consumer clientId=consumer-1, groupId=console-consumer-89327] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)


Now i am up the broker-id.0(server.properties), Leader was not change.
FYI,
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Sankar-Multibrokerapplication

Topic:Sankar-Multibrokerapplication     PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: Sankar-Multibrokerapplication    Partition: 0    Leader: 1       Replicas: 0,1,2 Isr: 1,2,0

Step:11
Basic Topic Operations like Modifying a Topic.

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Hello-Sankar-Kafka
Sankar-Multibrokerapplication
__consumer_offsets
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Hello-Sankar-Kafka
Topic:Hello-Sankar-Kafka        PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: Hello-Sankar-Kafka       Partition: 0    Leader: 0       Replicas: 0     Isr: 0
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic Hello-Sankar-Kafka --partitions 2
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic Hello-Sankar-Kafka
Topic:Hello-Sankar-Kafka        PartitionCount:2        ReplicationFactor:1     Configs:
        Topic: Hello-Sankar-Kafka       Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: Hello-Sankar-Kafka       Partition: 1    Leader: 1       Replicas: 1     Isr: 1
[root@sankar-devops kafka_2.12-1.1.0]#

Step:12
Deleting a Topic
[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Hello-Sankar-Kafka
Sankar-Multibrokerapplication
__consumer_offsets
[root@sankar-devops kafka_2.12-1.1.0]#

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic Hello-Sankar-Kafka
Topic Hello-Sankar-Kafka is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true

[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
Sankar-Multibrokerapplication
__consumer_offsets

PART1 - Step by step Kafka setup on CentOS.

FYI, ZooKeeper  need to setup before start the Kafka.

[root@sankar-devops opt]# wget http://redrockdigimark.com/apachemirror/kafka/1.1.0/kafka_2.12-1.1.0.tgz

[root@sankar-devops opt]# ls -ltrh
total 48M
-rw-r--r--.  1 root root  48M Mar 28 17:35 kafka_2.12-1.1.0.tgz
drwxr-xr-x.  3 root root   60 May 31 02:16 jdk
drwxr-xr-x. 11 1000 1000 4.0K May 31 03:27 zookeeper-3.4.12
[root@sankar-devops opt]#

[root@sankar-devops opt]# tar -xvzf kafka_2.12-1.1.0.tgz 

[root@sankar-devops opt]# ls -ltrh
total 48M
drwxr-xr-x.  6 root root   89 Mar 24 04:25 kafka_2.12-1.1.0
-rw-r--r--.  1 root root  48M Mar 28 17:35 kafka_2.12-1.1.0.tgz
drwxr-xr-x.  3 root root   60 May 31 02:16 jdk
drwxr-xr-x. 11 1000 1000 4.0K May 31 03:27 zookeeper-3.4.12

[root@sankar-devops opt]# cd kafka_2.12-1.1.0
[root@sankar-devops kafka_2.12-1.1.0]# ll
total 48
drwxr-xr-x. 3 root root  4096 Mar 24 04:25 bin
drwxr-xr-x. 2 root root  4096 Mar 24 04:25 config
drwxr-xr-x. 2 root root  4096 May 31 03:30 libs
-rw-r--r--. 1 root root 28824 Mar 24 04:21 LICENSE
-rw-r--r--. 1 root root   336 Mar 24 04:21 NOTICE
drwxr-xr-x. 2 root root    44 Mar 24 04:25 site-docs
[root@sankar-devops kafka_2.12-1.1.0]# cd bin
[root@sankar-devops bin]# ls -l
total 132
-rwxr-xr-x. 1 root root 1421 Mar 24 04:21 connect-distributed.sh
-rwxr-xr-x. 1 root root 1418 Mar 24 04:21 connect-standalone.sh
-rwxr-xr-x. 1 root root  861 Mar 24 04:21 kafka-acls.sh
-rwxr-xr-x. 1 root root  873 Mar 24 04:21 kafka-broker-api-versions.sh
-rwxr-xr-x. 1 root root  864 Mar 24 04:21 kafka-configs.sh
-rwxr-xr-x. 1 root root  945 Mar 24 04:21 kafka-console-consumer.sh
-rwxr-xr-x. 1 root root  944 Mar 24 04:21 kafka-console-producer.sh
-rwxr-xr-x. 1 root root  871 Mar 24 04:21 kafka-consumer-groups.sh
-rwxr-xr-x. 1 root root  948 Mar 24 04:21 kafka-consumer-perf-test.sh
-rwxr-xr-x. 1 root root  871 Mar 24 04:21 kafka-delegation-tokens.sh
-rwxr-xr-x. 1 root root  869 Mar 24 04:21 kafka-delete-records.sh
-rwxr-xr-x. 1 root root  863 Mar 24 04:21 kafka-log-dirs.sh
-rwxr-xr-x. 1 root root  862 Mar 24 04:21 kafka-mirror-maker.sh
-rwxr-xr-x. 1 root root  886 Mar 24 04:21 kafka-preferred-replica-election.sh
-rwxr-xr-x. 1 root root  959 Mar 24 04:21 kafka-producer-perf-test.sh
-rwxr-xr-x. 1 root root  874 Mar 24 04:21 kafka-reassign-partitions.sh
-rwxr-xr-x. 1 root root  868 Mar 24 04:21 kafka-replay-log-producer.sh
-rwxr-xr-x. 1 root root  874 Mar 24 04:21 kafka-replica-verification.sh
-rwxr-xr-x. 1 root root 7864 Mar 24 04:21 kafka-run-class.sh
-rwxr-xr-x. 1 root root 1376 Mar 24 04:21 kafka-server-start.sh
-rwxr-xr-x. 1 root root  997 Mar 24 04:21 kafka-server-stop.sh
-rwxr-xr-x. 1 root root  870 Mar 24 04:21 kafka-simple-consumer-shell.sh
-rwxr-xr-x. 1 root root  945 Mar 24 04:21 kafka-streams-application-reset.sh
-rwxr-xr-x. 1 root root  863 Mar 24 04:21 kafka-topics.sh
-rwxr-xr-x. 1 root root  958 Mar 24 04:21 kafka-verifiable-consumer.sh
-rwxr-xr-x. 1 root root  958 Mar 24 04:21 kafka-verifiable-producer.sh
-rwxr-xr-x. 1 root root 1722 Mar 24 04:21 trogdor.sh
drwxr-xr-x. 2 root root 4096 Mar 24 04:21 windows
-rwxr-xr-x. 1 root root  867 Mar 24 04:21 zookeeper-security-migration.sh
-rwxr-xr-x. 1 root root 1393 Mar 24 04:21 zookeeper-server-start.sh
-rwxr-xr-x. 1 root root 1001 Mar 24 04:21 zookeeper-server-stop.sh
-rwxr-xr-x. 1 root root  968 Mar 24 04:21 zookeeper-shell.sh
[root@sankar-devops bin]#

--------error---
[root@sankar-devops bin]# ./kafka-server-start.sh  config/server.properties
[2018-05-31 03:36:34,288] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2018-05-31 03:36:34,346] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.io.FileNotFoundException: config/server.properties (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at org.apache.kafka.common.utils.Utils.loadProps(Utils.java:510)
        at kafka.Kafka$.getPropsFromArgs(Kafka.scala:44)
        at kafka.Kafka$.main(Kafka.scala:81)
        at kafka.Kafka.main(Kafka.scala)

-----------------


[root@sankar-devops kafka_2.12-1.1.0]# bin/kafka-server-start.sh  config/server.properties
[2018-05-31 03:39:24,455] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2018-05-31 03:39:25,415] INFO starting (kafka.server.KafkaServer)
[2018-05-31 03:39:25,416] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2018-05-31 03:39:25,455] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2018-05-31 03:39:25,466] INFO Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.ZooKeeper)
[2018-05-31 03:39:25,467] INFO Client environment:host.name=sankar-devops.as.domcorp.zetainteractive.com (org.apache.zookeeper.ZooKeeper)
[2018-05-31 03:39:25,467] INFO Client environment:java.version=1.8.0_171 (org.apache.zookeeper.ZooKeeper)
[2018-05-31 03:39:25,467] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2018-05-31 03:39:25,467] INFO Client environment:java.home=/opt/jdk/jdk1


[root@sankar-devops ~]# jps
11064 QuorumPeerMain
11726 Kafka
12062 Jps

Above two daemons running on the terminal, where QuorumPeerMain is related to ZooKeeper daemon and another one is Kafka daemon.

PART2-Kafka

Step by step ZooKeeper setup on CentOS

[root@sankar-devops opt]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.12/zookeeper-3.4.12.tar.gz

[root@sankar-devops opt]# ls -ltrh
total 35M
-rw-r--r--. 1 root root 35M Apr 25 21:17 zookeeper-3.4.12.tar.gz
drwxr-xr-x. 3 root root  60 May 31 02:16 jdk

[root@sankar-devops opt]# tar -xvzf zookeeper-3.4.12.tar.gz 
[root@sankar-devops opt]# ls -l
total 35816
drwxr-xr-x.  3 root root       60 May 31 02:16 jdk
drwxr-xr-x. 10 1000 1000     4096 Mar 27 10:06 zookeeper-3.4.12
-rw-r--r--.  1 root root 36667596 Apr 25 21:17 zookeeper-3.4.12.tar.gz
[root@sankar-devops opt]# 

[root@sankar-devops opt]# cd zookeeper-3.4.12

[root@sankar-devops zookeeper-3.4.12]# mkdir data

Create config file::

[root@sankar-devops conf]# pwd
/opt/zookeeper-3.4.12/conf
[root@sankar-devops conf]# vi zoo.cfg

tickTime=2000
dataDir=/opt/zookeeper-3.4.12/data
clientPort=2181
initLimit=5
syncLimit=2

[root@sankar-devops zookeeper-3.4.12]# cd bin

[root@sankar-devops bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.12/bin/../conf/zoo.cfg
grep: /opt/zookeeper-3.4.12/bin/../conf/zoo.cfg: No such file or directory
mkdir: cannot create directory ‘’: No such file or directory
Starting zookeeper ... STARTED

note: i am executed zookeeper in wrong-path. 

[root@sankar-devops bin]# tail zookeeper.out 
2018-05-31 02:41:43,330 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
2018-05-31 02:41:43,330 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=3.10.0-514.el7.x86_64
2018-05-31 02:41:43,330 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=root
2018-05-31 02:41:43,330 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=/root
2018-05-31 02:41:43,331 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/opt/zookeeper-3.4.12/bin
2018-05-31 02:41:43,347 [myid:] - INFO  [main:ZooKeeperServer@835] - tickTime set to 2000
2018-05-31 02:41:43,347 [myid:] - INFO  [main:ZooKeeperServer@844] - minSessionTimeout set to -1
2018-05-31 02:41:43,347 [myid:] - INFO  [main:ZooKeeperServer@853] - maxSessionTimeout set to -1
2018-05-31 02:41:43,381 [myid:] - INFO  [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2018-05-31 02:41:43,404 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
[root@sankar-devops bin]# 

[root@sankar-devops bin]# netstat -ntlup |grep 2181
tcp6       0      0 :::2181                 :::*                    LISTEN      9442/java           
[root@sankar-devops bin]# 

[root@sankar-devops bin]# ./zkCli.sh 
Connecting to localhost:2181
2018-05-31 02:43:32,917 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
................................
Welcome to ZooKeeper!
JLine support is enabled
......................................
WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0] 
[zk: localhost:2181(CONNECTED) 0] 

[root@sankar-devops bin]# ./zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.12/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
[root@sankar-devops bin]# 


[root@sankar-devops ~]# jps
11064 QuorumPeerMain
11726 Kafka

12062 Jps

Note: where QuorumPeerMain is related to ZooKeeper daemon 

Steps how to clear ESM Log size is full ?

Embedded System Management (ESM) Log

When we open Dell OMSA, we've observed that hardware log is critical status and evenlogs are filled saying that ESM log is full.

So we need to clear the ESM logs like two ways,

1. we can clear event-log though Dell OMSA UI
2. We can clear event-log though Server CLI


1. we can clear event-log though Dell OMSA UI

Login into Dell OMSA --->go to Logs tab ---> Click on hardware ---> Log cleared

2.
[root@sankar~]# omconfig system esmlog action=clear
Embedded System Management (ESM) log cleared successfully.

[root@sankar ~]#

Step by Step Installation process of JAVA 8 on RHEL7/CentOS7

[root@sankar-devops ~]# cd /opt
[root@sankar-devops opt]#
[root@sankar-devops opt]# pwd
/opt
[root@sankar-devops opt]#

[root@sankar-devops opt]# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u171-b11/512cd62ec5174c3487ac17c61aaa89e8/jdk-8u171-linux-x64.tar.gz"

[root@sankar-devops opt]# ls -l
total 186420
-rw-r--r--. 1 root root 190890122 Apr  3 23:35 jdk-8u171-linux-x64.tar.gz
[root@sankar-devops opt]#

[root@sankar-devops opt]# tar -xvzf jdk-8u171-linux-x64.tar.gz 

[root@sankar-devops opt]# ls -ltrh
total 183M
drwxr-xr-x. 8   10  143  255 Mar 29 05:48 jdk1.8.0_171
-rw-r--r--. 1 root root 183M Apr  3 23:35 jdk-8u171-linux-x64.tar.gz

[root@sankar-devops opt]# mkdir jdk

[root@sankar-devops opt]# mv jdk1.8.0_171 /opt/jdk/

[root@sankar-devops jdk1.8.0_171]# pwd
/opt/jdk/jdk1.8.0_171


[root@sankar-devops jdk1.8.0_171]# vi ~/.bashrc 


export JAVA_HOME=/opt/jdk/jdk1.8.0_171
export PATH=$PATH:$JAVA_HOME/bin
:wq!

[root@sankar-devops jdk1.8.0_171]# source  ~/.bashrc 
[root@sankar-devops jdk1.8.0_171]#

[root@sankar-devops jdk1.8.0_171]# update-alternatives --install /usr/bin/java java /opt/jdk/jdk1.8.0_171/bin/java 100
[root@sankar-devops jdk1.8.0_171]#

[root@sankar-devops jdk1.8.0_171]# java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
[root@sankar-devops jdk1.8.0_171]# 

Steps for RHEL5 to OLE5 Migration.

[root@sankar-devops ~]#  cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.10 (Tikanga)

[root@sankar-devops ~]# cat /etc/issue
Red Hat Enterprise Linux Server release 5.10 (Tikanga)
Kernel \r on an \m


[root@sankar-devops ~]#
[root@sankar-devops ~]# yum -y remove rhn-client-tools yum-rhn-plugin up2date subscription-manager
Loaded plugins: rhnplugin, security
There was an error communicating with RHN.
RHN Satellite or RHN Classic support will be disabled.

Setting up Remove Process
No Match for argument: up2date
No Match for argument: subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package rhn-client-tools.noarch 0:0.4.20-77.el5 set to be erased
--> Processing Dependency: rhn-client-tools = 0.4.20-77.el5 for package: rhn-setup
--> Processing Dependency: rhn-client-tools = 0.4.20-77.el5 for package: rhn-check
---> Package yum-rhn-plugin.noarch 0:0.5.4-26.el5 set to be erased
--> Running transaction check
---> Package rhn-check.noarch 0:0.4.20-77.el5 set to be erased
--> Processing Dependency: rhn-check >= 0.0.8 for package: rhnsd
---> Package rhn-setup.noarch 0:0.4.20-77.el5 set to be erased
--> Running transaction check
---> Package rhnsd.x86_64 0:4.7.0-10.el5 set to be erased
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================
 Package                             Arch                      Version                             Repository                    Size
======================================================================================================================================
Removing:
 rhn-client-tools                    noarch                    0.4.20-77.el5                       installed                     0.0
 yum-rhn-plugin                      noarch                    0.5.4-26.el5                        installed                    182 k
Removing for dependencies:
 rhn-check                           noarch                    0.4.20-77.el5                       installed                     0.0
 rhn-setup                           noarch                    0.4.20-77.el5                       installed                     0.0
 rhnsd                               x86_64                    4.7.0-10.el5                        installed                     72 k

Transaction Summary
======================================================================================================================================
Remove        5 Package(s)
Reinstall     0 Package(s)
Downgrade     0 Package(s)

Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Erasing        : rhn-setup                                                                                                      1/5
  Erasing        : rhnsd                                                                                                          2/5
  Erasing        : rhn-check                                                                                                      3/5
  Erasing        : rhn-client-tools                                                                                               4/5
warning: /etc/sysconfig/rhn/up2date saved as /etc/sysconfig/rhn/up2date.rpmsave
  Erasing        : yum-rhn-plugin                                                                                                 5/5
warning: /etc/yum/pluginconf.d/rhnplugin.conf saved as /etc/yum/pluginconf.d/rhnplugin.conf.rpmsave

Removed:
  rhn-client-tools.noarch 0:0.4.20-77.el5                             yum-rhn-plugin.noarch 0:0.5.4-26.el5                         

Dependency Removed:
  rhn-check.noarch 0:0.4.20-77.el5              rhn-setup.noarch 0:0.4.20-77.el5              rhnsd.x86_64 0:4.7.0-10.el5           

Complete!
[root@sankar-devops ~]#

[root@sankar-devops ~]# cd /etc/yum.repos.d

[root@sankar-devops yum.repos.d]#rm -rf *

[root@sankar-devops yum.repos.d]# ll
total 0
[root@sankar-devops yum.repos.d]# wget http://yum.oracle.com/public-yum-el5.repo
--2018-05-09 00:44:08--  http://yum.oracle.com/public-yum-el5.repo
Resolving yum.oracle.com... 184.25.162.159
Connecting to yum.oracle.com|184.25.162.159|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4886 (4.8K) [text/plain]
Saving to: `public-yum-el5.repo'

100%[============================================================================================>] 4,886       --.-K/s   in 0s   

2018-05-09 00:44:09 (12.5 MB/s) - `public-yum-el5.repo' saved [4886/4886]

[root@sankar-devops yum.repos.d]# wget https://linux-update.oracle.com/rpms/ULN-CA-CERT.sha2 -O /usr/share/rhn/ULN-CA-CERT
--2018-05-09 00:44:16--  https://linux-update.oracle.com/rpms/ULN-CA-CERT.sha2
wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5 -O /etc/pki/rpm-gpg/RPM-GPG-KEY-oracleResolving linux-update.oracle.com... 137.254.56.42
Connecting to linux-update.oracle.com|137.254.56.42|:443... connected.
HTTP request sent, awaiting response...

[root@sankar-devops yum.repos.d]# ll
total 8
-rw-r--r-- 1 root root 4886 Feb 12 20:32 public-yum-el5.repo

[root@sankar-devops yum.repos.d]# wget https://linux-update.oracle.com/rpms/ULN-CA-CERT.sha2 -O /usr/share/rhn/ULN-CA-CERT
--2018-05-09 00:47:43--  https://linux-update.oracle.com/rpms/ULN-CA-CERT.sha2
Resolving linux-update.oracle.com... 137.254.56.42
Connecting to linux-update.oracle.com|137.254.56.42|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7727 (7.5K) [text/plain]
Saving to: `/usr/share/rhn/ULN-CA-CERT'

100%[============================================================================================>] 7,727       11.4K/s   in 0.7s 

2018-05-09 00:49:50 (11.4 KB/s) - `/usr/share/rhn/ULN-CA-CERT' saved [7727/7727]

[root@sankar-devops yum.repos.d]# wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5 -O /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
--2018-05-09 00:52:11--  http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
Resolving public-yum.oracle.com... 184.84.98.170
Connecting to public-yum.oracle.com|184.84.98.170|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1397 (1.4K) [text/plain]
Saving to: `/etc/pki/rpm-gpg/RPM-GPG-KEY-oracle'

100%[============================================================================================>] 1,397       --.-K/s   in 0s   

2018-05-09 00:52:12 (88.8 MB/s) - `/etc/pki/rpm-gpg/RPM-GPG-KEY-oracle' saved [1397/1397]

[root@sankar-devops yum.repos.d]# gpg --quiet --with-fingerprint /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle

pub  1024D/1E5E0159 2007-05-18 Oracle OSS group (Open Source Software group) <build@oss.oracle.com>
      Key fingerprint = 99FD 2766 28EE DECB 5E5A  F5F8 66CE D3DE 1E5E 0159
sub  1024g/D303656F 2007-05-18 [expires: 2015-05-16]

[root@sankar-devops yum.repos.d]# yum -y upgrade yum* enterprise-release-notes oracle-logos rpm* redhat* system-config*
Loaded plugins: security
el5_latest                                                                                                     | 1.4 kB     00:00   
el5_latest/primary                                                                                             |  29 MB     05:29   
el5_latest                                                                                                                15734/15734
ol5_UEK_latest                                                                                                 | 1.2 kB     00:00

[root@sankar-devops yum.repos.d]# yum update

[root@sankar-devops yum.repos.d]# cat /etc/issue

Oracle Linux Server release 5.11

Kernel \r on an \m