Apache Kafka Tutorial

The Apache Kafka open source software is one of the best solutions for storing and pro­cessing data streams. This messaging and streaming platform, which is licensed under Apache 2.0, features fault tolerance, excellent scalab­il­ity, and a high read and write speed. These factors, which are highly appealing for Big Data ap­plic­a­tions, are based on a cluster which allows dis­trib­uted data storage and rep­lic­a­tion. There are four different in­ter­faces for com­mu­nic­at­ing with the cluster with a simple TCP protocol serving as the basis for com­mu­nic­a­tion.

This Kafka tutorial explains how to get started with the Scala-based ap­plic­a­tion, beginning with in­stalling Kafka and the Apache ZooKeeper software required to use it.

To run a powerful Kafka cluster, you will need the right hardware. The de­vel­op­ment team re­com­mends using quad-core Intel Xeon machines with 24 gigabytes of memory. It is essential that you have enough memory to always cache the read and write accesses for all ap­plic­a­tions that actively access the cluster. Since Apache Kafka’s high data through­put is one of its draws, it is crucial to choose a suitable hard drive. The Apache Software Found­a­tion re­com­mends a SATA hard drive (8 x 7200 RPM). When it comes to avoiding per­form­ance bot­tle­necks, the following principle applies: the more hard drives, the better.

In terms of software, there are also some re­quire­ments that must be met in order to use Apache Kafka for managing incoming and outgoing data streams. When choosing your operating system, you should opt for a Unix operating system, such as Solaris, or a Linux dis­tri­bu­tion. This is because Windows platforms receive limited support. Apache Kafka is written in Scala, which compiles to Java bytecode, so you will need the latest version of the Java SE De­vel­op­ment Kits (JDK) installed on your system. This also includes the Java Runtime En­vir­on­ment, which is required for running Java ap­plic­a­tions. You will also need the Apache ZooKeeper service, which syn­chron­ises dis­trib­uted processes.

For an ex­plan­a­tion of what software is required, see the previous part of this Kafka tutorial. If it is not already installed on your system, we recommend in­stalling the Java Runtime En­vir­on­ment first. Many newer versions of Linux dis­tri­bu­tions, such as Ubuntu, which is used as an example operating system in this Apache Kafka tutorial (Version 17.10), already have OpenJDK, a free im­ple­ment­a­tion of JDK, in their official package re­pos­it­ory. This means that you can easily install the Java De­vel­op­ment Kit via this re­pos­it­ory by typing the following command into the terminal:

Im­me­di­ately after in­stalling Java, install the Apache ZooKeeper process syn­chron­isa­tion service. The Ubuntu package re­pos­it­ory also provides a ready-to-use package for this service, which can be executed using the following command line:

You can then use an ad­di­tion­al command to check whether the ZooKeeper service is active:

If the syn­chron­isa­tion service is not running, you can start it at any time using this command:

To ensure that ZooKeeper will always launch auto­mat­ic­ally at startup, add an autostart entry at the end:

Finally, create a user profile for Kafka, which you will need to use the server later. To do so, open the terminal again and enter the following command:

Using the passwd password manager, you can then add a password to the user by typing the following command followed by the desired password:

Next, grant the user “kafka” sudo rights:

You can now log in at any time with the newly-created user profile:

We have arrived at the point in this tutorial where it is time to download and install Kafka. There are a number of trusted sources where you can download both older and current versions of the data stream pro­cessing software. For example, you can obtain the in­stall­a­tion files directly from the Apache Software Found­a­tion’s download directory. It is highly re­com­men­ded that you work with a current version of Kafka, so you may need to adjust the following download command before entering it into the terminal:

Since the down­loaded file will be com­pressed, you will then need to unpack it:

Use the --strip 1 flag to ensure that the extracted files are saved directly to the ~/kafka directory. Otherwise, based on the version used in this Kafka tutorial, Ubuntu would store all files in the ~/kafka/kafka_2.12-2.1.0 directory. To do this, you must have pre­vi­ously created a directory named “kafka” using mkdir and switched to it (via “cd kafka”).

Now that you have installed Apache Kafka, the Java Runtime En­vir­on­ment and ZooKeeper, you can run the Kafka service at any time. Before you do this, however, you should make a few small ad­just­ments to its con­fig­ur­a­tions so that the software is optimally con­figured for its upcoming tasks.

Kafka does not allow you to delete topics (i.e. the storage and cat­egor­isa­tion com­pon­ents in a Kafka cluster) in its default set-up. However, you can easily change this by using the server.prop­er­ties Kafka con­fig­ur­a­tion file. To open this file, which is located in the config directory, use the following terminal command in the default text editor nano:

At the end of this con­fig­ur­a­tion file, add a new entry, which enables you to delete topics:

The next step in the Kafka tutorial is to create unit files for ZooKeeper and Kafka that allow you to perform common actions such as starting, stopping and re­start­ing the two services in a manner con­sist­ent with other Linux services. To do so, you need to create and set up .service files for the systemd session manager for both ap­plic­a­tions.

First, create the file for the ZooKeeper syn­chron­isa­tion service by entering the following command in the terminal:

This will not only create the file but also open it in the nano text editor. Now, enter the following lines and then save the file.

As a result, systemd will un­der­stand that ZooKeeper requires the network and the file system to be ready before it can start. This is defined in the [Unit] section. The [Service] section specifies that the session manager should use the files zookeeper-server-start.sh and zookeeper-server-stop.sh to start and stop ZooKeeper. It also specifies that ZooKeeper should be restarted auto­mat­ic­ally if it stops un­ex­pec­tedly. The [Install] entry controls when the file is started using ".multi-user.target” as the default value for a multi-user system (e.g. a server).

To create the .service file for Apache Kafka, use the following terminal command:

Then, copy the following content into the new file that has already been opened in the nano text editor:

Once you have suc­cess­fully created the session manager entries for Kafka and ZooKeeper, you can start Kafka with the following command:

By default, the systemd program uses a central protocol or journal in which all log messages are auto­mat­ic­ally written. As a result, you can easily check whether the Kafka server has been started as desired:

If you have suc­cess­fully started Apache Kafka manually, enable finish by ac­tiv­at­ing automatic start during system boot:

This part of the Kafka tutorial involves testing Apache Kafka by pro­cessing an initial message using the messaging platform. To do so, you need a producer and a consumer (i.e. an instance which enables you to write and publish data to topics and an instance which can read data from a topic). First of all, you need to create a topic, which in this case should be called Tu­tori­alTop­ic. Since this is a simple test topic, it should only contain a single partition and a single replica:

Next, you need to create a producer that adds the first example message "Hello, World!" to the newly created topic. To do so, use the kafka-console-producer.sh shell script, which needs the Kafka server’s host name and port (in this example, Kafka’s default path) as well as the topic name as arguments:

Next, using the kafka-console-consumer.sh script, create a Kafka consumer that processes and displays messages from Tu­tori­alTop­ic. You will need the Kafka server’s host name and port as well as the topic name as arguments. In addition, the “--from-beginning” argument is attached so that the consumer can actually process the “Hello, World!” message, which was published before the consumer was created: