SNAS Kafka Docker Install Steps
Kafka container is a pre-configured install for SNAS. You can use this container for SNAS or you can use your own Kafka install. SNAS supports consumer load balancing, so we do define at least 4 partitions.
- Apache Kafka 0.10.1.1 - High performing message bus (listening ports are TCP 2181 and 9092)
Recommended Current Linux Distributions
- Ubuntu 16.04/Xenial
- CentOS 7/RHEL 7
1) Install docker
Docker host should be Linux x86_64. Follow the Docker Instructions to install docker.
2) Download the docker image
docker pull openbmp/kafka
3) Create Kafka persistent volume
Depending on your docker devicedriver, the root filesystem in the container may not be the best place to house the kafka data files. The default for devicemapper is 10GB, which isn’t enough disk space for a large data collection. The way to work around this is to use the below docker volume.
mkdir -p /var/openbmp/kafka chmod 777 /var/openbmp/kafka
The mode of 777 can be changed to chown
but you’ll have to get that ID by looking at the file owner after starting the container.
4) Run docker container
Below table lists the environment variables that can be used with
docker -e <name=value>
|KAFKA_FQDN||hostname||REQUIRED. Fully qualified hostname that can be resolved inside docker container (e.g.
• You MUST define the KAFKA_FQDN as a ‘hostname’ that can be resolved inside the docker container.
• We recommend to set it to ‘localhost’ (or ‘127.0.0.1’) if you are not planning to have your own clients (consumers or producers) outside this container.
• KAFKA_FQDN is used by Kafka to advertise the leader (advertised.host.name) which handles all read and write requests for a partition. If it can not be resolved, there will be no messages published or consumed (without a clear error message in the logs).
• If you are planning to have your own clients outside the container that need access to Kafka running inside the docker container, then the ‘hostname’ must be resolvable inside the container as well as on the hosts where the container and the clients are running.
docker run -d \ --name=openbmp_kafka \ -e KAFKA_FQDN=localhost \ -v /var/openbmp/kafka:/data/kafka \ -p 2181:2181 -p 9092:9092 \ openbmp/kafka
You can navigate all the log files from within the container. Connect to container using:
docker exec -it openbmp_kafka bash
Or, you can use standard docker exec commands on host to monitor the log files. To monitor logs, use:
docker exec openbmp_kafka tail -f /var/log/*.log
Or, you can monitor the docker container by getting the console logs. This is useful if the container exits due to invalid start or for another reason. To see console logs for collector, use:
docker logs openbmp_kafka
You can also monitor individual Kafka topics to see messages. For example, you can monitor the openbmp.parsed.unicast_prefix topic using:
docker exec openbmp_kafka /usr/local/kafka/bin/kafka-console-consumer.sh -z localhost --topic openbmp.parsed.unicast_prefix
System Start/Restart Config (Ubuntu 16.04/Xenial)
By default, the containers will not start automatically on system boot/startup. You can use the below example to instruct the container to start automatically.
You can read more at Docker Admin Guide on how to start containers automatically.
--name=openbmp_kafkaparameter given to the
docker runcommand is used with the
-a openbmp_kafkaparameter below to start the container by name instead of container ID. You can use whatever name you want, but make sure to use the same name used in docker run.
cat <<END > /etc/init/kafka-openbmp.conf description "OpenBMP Kafka container" author "firstname.lastname@example.org" start on filesystem and started docker stop on runlevel [!2345] respawn script /usr/bin/docker start -a openbmp_kafka end script END