In this blog post, we will install Apache Kafka, create a topic, and send a few messages.

Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.

1. Install Kafka

The easiest way to install Apache Kafka on macOS is by using Homebrew.

brew install kafka

2. Start the Zookeper and Kafka server

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

Zookeeper is used to manage brokers in the Kafka cluster.

3. Create a topic

Kafka is architectured as distributed transaction log. This means that we can break down the data of each topic into multiple partitions which can be running on one or more servers all managed by the Kafka Cluster. In this case, we’re using one partition: --partitions 1.

For each topic, we can also set the --replication-factor where n is the number of servers. If we set n to 2, our topic’s partitions log will be replicated on two servers A and B. If there is a failure on the server A, automatic failover to the server B will happen.

Let’s create a topic named hello-world:

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello-world

We should see this message: Created topic "hello-world". If you’ve created many different topics, you can view them by using the list command:

kafka-topics --list --zookeeper localhost:2181

4. Sending Messages

At its most basic principle, we use Producers to publish messages to the specific topics in the Kafka Cluster, and Consumers to subscribe to the specific topics and retrieve the messages.

Producer

We can have an arbitrary number of processes called producers. When we execute the following command, we will start command line client that will be listening for any input. In my example, I’ll be sending messages a, b, c, d, e, and f to the topic hello-world.

kafka-console-producer --broker-list localhost:9092 --topic hello-world

asciicast

Consumer

Consumers role is to query the messages. We have published 6 messages so far to the topic helo-world. Let’s see them:

kafka-console-consumer --bootstrap-server localhost:9092 --topic hello-world --from-beginning

Consumer client will now be listening for any new messages that might be published by the producer. You can see that I’m using --from-beginning flag. It will make sure that messages from the beginning of the stream are retrieved.

asciicast

Happy message passing!