Kafka Essentials — A Practical Beginner’s Guide

Jason Yang
26 Mar, 2026
Views —

When you start working with microservices, one question eventually comes up:

“How should services communicate with each other?”

Calling each other directly via REST APIs is simple at first. But as the number of services grows, dependencies become tangled, and a single failure can trigger cascading issues.

That’s where many teams turn to Kafka.

However, Kafka can feel intimidating at first because of unfamiliar terms like Topic, Partition, Consumer Group, ACK, and DLT.

In this guide, I’ll walk through the core concepts you actually need to understand Kafka in real-world systems — in the simplest way possible.

1. What is Kafka?

Kafka is a distributed event streaming platform.

In simple terms, it acts as a durable message storage and delivery system between services.

Producer ── publish ──> [ Kafka ] ── deliver ──> Consumer

Unlike traditional message queues (like RabbitMQ), Kafka does not delete messages immediately after they are consumed.

This allows:

Multiple consumers to read the same message independently
Reprocessing of past messages

2. Core Components

Broker

A Broker is a Kafka server.

It:

Stores messages on disk
Handles requests from producers and consumers

[ Broker 1 ]  [ Broker 2 ]  [ Broker 3 ]
      └──────────┴──────────┘
          Kafka Cluster

Development: usually 1 broker is enough
Production: multiple brokers for high availability

Topic

A Topic is a category or channel for messages.

Think of it like a folder or label.

myapp.user.account.created
myapp.report.file.created
myapp.order.created

Producers write to topics
Consumers subscribe to topics

Partition

A Partition is a physical subdivision of a topic.

Topic: myapp.user.account.created
  ├── Partition 0: [msg1] [msg4] [msg7]
  ├── Partition 1: [msg2] [msg5] [msg8]
  └── Partition 2: [msg3] [msg6] [msg9]

Why partitions matter:

Enable parallel processing
Improve throughput

Partition Key

How messages are distributed:

No key → round-robin (no ordering guarantee)
With key → same key → same partition → ordering guaranteed

That’s why userId is often used as a partition key — it preserves the order of events for each user.

Offset

An Offset is the position of a message within a partition.

Partition 0: [0] [1] [2] [3] ...
                          ↑
                   last processed

Consumers:

Track their current offset
Commit offsets to Kafka

This allows them to:

Resume after restart
Avoid reprocessing

3. Producer

A Producer publishes messages to Kafka.

kafkaTemplate.send(topic, key, value);

await producer.send_and_wait(topic, value=payload, key=key)

Key Producer Settings

Setting	Description
`acks=all`	Ensures all brokers confirm write (safe)
`enable.idempotence=true`	Prevents duplicate messages
`retries=3`	Retry on failure

4. Consumer & Consumer Group

Consumer

A Consumer reads messages from Kafka.

Subscribes to topics
Processes messages sequentially

Consumer Group

A Consumer Group is a set of consumers working together.

Topic (3 partitions)

Consumer Group:
  A → Partition 0
  B → Partition 1
  C → Partition 2

Key rules:

Each partition is handled by only one consumer per group
More consumers than partitions → some stay idle

Multiple Consumer Groups

Different groups can read the same data independently:

Group A → data processing
Group B → analytics

Each group maintains its own offsets.

auto.offset.reset

Determines where to start reading:

Value	Behavior
`earliest`	Read from beginning (recommended)
`latest`	Only new messages

5. ACK (Acknowledgment)

ACK means confirming that a message has been successfully processed.

Auto Commit (Risky)

Read message
→ auto commit
→ processing fails ❌
→ message lost

Manual ACK (Recommended)

Read message
→ process
→ success → commit
→ failure → retry

Example:

ack.acknowledge();

👉 In production, manual ACK is essential.

6. Dead Letter Topic (DLT)

DLT handles messages that repeatedly fail.

Original Topic
→ processing fails
→ retry (3 times)
→ still fails
→ move to DLT

Without DLT:

One bad message blocks everything

With DLT:

Problem messages are isolated
Normal processing continues

Typical usage:

Logging
Alerts (Slack, email)
Manual reprocessing

7. KRaft Mode (No ZooKeeper)

Older Kafka required ZooKeeper.

Now, with Kafka 3.3+:

👉 KRaft mode removes the need for ZooKeeper

Before: Kafka + ZooKeeper
Now:    Kafka only

Example config:

KAFKA_PROCESS_ROLES: broker,controller

One node handles:

Message storage
Cluster metadata

8. Delivery Guarantees

Kafka supports three delivery models:

Type	Description	Duplicate	Loss
At-most-once	No retry	❌	✅
At-least-once	Retry on failure	✅	❌
Exactly-once	Perfect delivery	❌	❌

👉 In practice:
At-least-once is most common

So you must handle duplicates using:

idempotencyKey

9. Summary

Kafka Broker
 └── Topic
     └── Partitions
         └── Messages (Offsets)

Concept	Description
Broker	Kafka server
Topic	Message category
Partition	Parallel unit
Offset	Message position
Producer	Sends messages
Consumer	Reads messages
Consumer Group	Parallel processing group
ACK	Processing confirmation
DLT	Failed message storage
KRaft	ZooKeeper-free Kafka

Final Thoughts

Kafka may look complex at first, but once you understand these core concepts, everything starts to make sense.

In real systems, Kafka helps you:

Reduce service coupling
Improve reliability
Handle large-scale data flow

If you’re working with microservices, it’s no longer optional — it’s a powerful tool worth mastering.

#Kafka