Apache Kafka: From Message Queues to Distributed Event Streaming
Nihal Gupta
Apache Kafka is a message queue widely used to scale backend systems. In a message queue a producer puts messages in a queue and a consumer pops out those messages and performs operations on it.
Why do we need it ?
Suppose we have an ecommerce app with an endpoint to buy a product : POST /buy-product when the client hits this api , multiple operations are triggered like : payment, update inventory, send notifications, etc. We could do this things synchronously, but the response would take longer (like >= 2secs) , we don’t want that, so we let these task handled later by some worker .
You can see that some of these tasks are non critical like , sending notification , we can afford some delay in it. So we do these non critical tasks asynchronously with delay.
so finally when /buy-product is hit, we do tasks in this order: 1 -> payment 2 -> update inventory 3 -> order creation 3 -> push notification related details into kafka 4 -> send response OK
since the critical operations must be done before we confirm the purchase we do it synchronously we didn’t do the non critical task immediately , instead we pushed it into kafka. for future , by doing this we reduce our response time significantly from >= 2secs to 100 ms we delegate the non critical task to some consumer which will consume this queue and process it.
so now we understand when to use kafka : when there are multiple simultaneous requests we use a queue to put non critical operations and process them later.
Now let’s understand the internals of Kafka
What is Kafka ?
At a high level, Kafka is a distributed event streaming platform. An application can publish events to Kafka, and other applications can consume those events.
Producer : A producer is an application that publishes records to Kafka. For example, after an order is created: { “orderId”: 12345, “userId”: 987, “amount”: 2499, “…” : “..” } the order service can publish this event to kafka.
Consumer : consumer is an application which subscribes to the kafka topic and process the messages. For example: Kafka | v Notification Service
The notification service can consume the OrderCreated event and send an email.Another service can consume the same event for analytics. Topic : Kafka stores records inside topics. You can think of a topic as a named stream of events. For example: order-events, payment-events, user-events, location-events Our order service might publish to: order-events while multiple consumers subscribe to that topic.
Broker : A Kafka broker is a Kafka server. A Kafka cluster usually consists of multiple brokers. For example: Kafka Cluster : [Broker 1 Broker 2 Broker 3] These brokers work together to store and serve Kafka data.
Partitions : A Kafka topic is divided into partitions. For example:
order-events [ Partition 0, Partition 1, Partition 2, Partition 3] Each partition is an ordered sequence of records.
Conceptually:
Partition 0: 0 -> 1 -> 2 -> 3 -> 4 -> 5 Partition 1: 0 -> 1 -> 2 -> 3 -> 4 Consumer Groups : Now imagine we have 4 partitions and 3 consumers.
Partitions: P0 P1 P2 P3 Consumers: C1 C2 C3
Kafka assigns partitions among the consumers. For example:
C1 — -> P0 C2 — -> P1 C3 — -> P2, P3
If a consumer leaves the group, Kafka can rebalance the partitions among the remaining consumers. This is called a consumer group rebalance.
What if we add more consumers?
Suppose we have: 4 partitions 6 consumers
We cannot actively process one partition with multiple consumers from the same consumer group at the same time.
So we might get:
C1 — -> P0 C2 — -> P1 C3 — -> P2 C4 — -> P3 C5 — -> idle C6 — -> idle
This is why the number of partitions puts a limit on the amount of parallelism we can achieve within a consumer group. If you want more consumers to work in parallel, you generally need enough partitions.
This was all about the theory, there are too many things which I haven’t covered here like how rebalancing happens internally , replicas, leader nodes, retention period, consumer offsets and many more. I’d suggest you to implement kafka in your projects and play with it to understand it better.
Documentation for getting started :