three server cluster for fail redundancy

each server will be a broker server with a guest OS containing Bins/Libs

Each server will have Java, Kafka and Zookeeper running

1. wget kafka tar file and gzip it into /opt/kafka
2. apt/yum install -y default-jdk = install java developer kit
3. java -version = verify java was installed
4. swapoff -a = to dsiable RAM swap
5. sudo sed -i '/ swap / s/^/#/' /etc/fstab = comment out swap in the /etc/fstab file

create a new DIR and then CHOWN -R for Kafka and Zookeeper

sudo mkdir -p /data/kafka = for message logs

sudo mkdir -p /data/zookeeper = for snapshots

Specify ID for each zookeeper server

1. echo "1" > /data/zookeeper/myid = creates a file in location called myid with the contents “1”

Modify kafka and zookeeper config files

# change this for each broker
broker.id=[broker_number]
# change this to the hostname of each broker
advertised.listeners=PLAINTEXT://[hostname]:9092
# The ability to delete topics
delete.topic.enable=true
# Where logs are stored
log.dirs=/data/kafka
# default number of partitions
num.partitions=8
# default replica count based on the number of brokers
default.replication.factor=3
# to protect yourself against broker failure
min.insync.replicas=2
# logs will be deleted after how many hours
log.retention.hours=168
# size of the log files
log.segment.bytes=1073741824
# check to see if any data needs to be deleted
log.retention.check.interval.ms=300000
# location of all zookeeper instances and kafka directory
zookeeper.connect=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/kafka
# timeout for connecting with zookeeper
zookeeper.connection.timeout.ms=6000
# automatically create topics
auto.create.topics.enable=true

rm /opt/kafka/config/server.properties

vim /opt/kafka/config/server.properties

replace broker_number with brokerid(myid) and advertised listeners(hostname)