What are topics and partitions in Apache Kafka?

Vaibhav Kothia May 26, 2023 0 Comments

Topics: A topic is a category or feed name to which records are published. Each record consists of a key, a value, and a timestamp. Examples of topics include “user-signups”, “page-views”, and “error-logs”.

Partitions: A partition is a unit of parallelism in Kafka. It is an ordered, immutable sequence of records that is continually appended to. A partition is identified by its topic and partition number. For example, the topic “page-views” may have four partitions labelled 0, 1, 2, and 3. Each partition can be stored on a different machine to allow for multiple consumers to read from a topic in parallel.

What are the main components of Apache Kafka?

Vaibhav Kothia May 26, 2023 0 Comments

1. Brokers: A Kafka cluster consists of one or more servers (Kafka brokers) running Kafka. Each broker is identified by its id, and it contains certain topic partitions. For example, a broker with id 1 may contain topic partitions 0 and 1.

2. Topics: A topic is a category or feed name to which messages are published. For example, a topic can be a user activity log or a financial transaction log.

3. Producers: Producers are processes that publish data to topics. For example, a producer may publish a user purchase event to a topic called “user_purchases”.

4. Consumers: Consumers are processes that subscribe to topics and process the published messages. For example, a consumer may subscribe to the “user_purchases” topic and process each message to update the user’s profile in the database.

5. Zookeeper: Apache Zookeeper is a distributed coordination service that helps maintain configuration information and provide synchronization across the cluster. It is used by Kafka to manage the cluster.