Kafka Setup & Topic Design — A Practical Local Development Guide

Jason Yang
26 Mar, 2026
Views —

This is a follow-up to the previous post:
Kafka Essentials — A Practical Beginner’s Guide

If terms like Topic, Partition, Consumer Group, ACK, or DLT still feel unfamiliar, I recommend reading that first.

Once you understand the concepts, the next step is to set up a working environment.

In this post, we’ll:

Set up Kafka locally using Docker Compose
Define practical topic naming conventions
Establish simple design rules you can reuse in real projects

This setup will be used later with both Spring Boot and FastAPI examples.

1. Infrastructure Setup

Local Kafka for Development

For local development, you can run Kafka on a shared machine (or your laptop) using Docker.
All services connect through an external listener.

Item	Value
Broker	`<KAFKA_HOST_IP>:9092`
UI	`http://<KAFKA_HOST_IP>:8989`
Auth	PLAINTEXT (dev only)
Mode	KRaft (no ZooKeeper)

Docker Compose

# kafka-compose.yml
services:
  kafka:
    container_name: app-kafka
    image: apache/kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller

      KAFKA_LISTENERS: PLAINTEXT://:9090,CONTROLLER://:9091,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9090,EXTERNAL://<KAFKA_HOST_IP>:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9091
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"

      CLUSTER_ID: <generated-uuid>

    volumes:
      - ./volumes:/var/lib/kafka/data
    restart: unless-stopped

    healthcheck:
      test: ["CMD", "/opt/kafka/bin/kafka-topics.sh", "--bootstrap-server", "localhost:9090", "--list"]
      interval: 15s
      timeout: 10s
      start_period: 30s
      retries: 5

  kafka-ui:
    container_name: app-kafka-ui
    image: provectuslabs/kafka-ui:latest
    ports:
      - "8989:8080"
    environment:
      - KAFKA_CLUSTERS_0_NAME=local-cluster
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9090
    depends_on:
      kafka:
        condition: service_healthy
    restart: unless-stopped

Listener Structure (Important)

Kafka uses different listeners depending on where the request comes from:

PLAINTEXT://:9090 → internal Docker network (e.g., kafka-ui)
EXTERNAL://:9092 → external clients (your local services)
CONTROLLER://:9091 → internal KRaft controller

Topics will be automatically created when you publish your first message:

KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"

2. Topic Design

Naming Convention

A simple and scalable format:

{app}.{service}.{entity}.{action}

Examples:

myapp.user.account.created
myapp.report.file.deleted
myapp.order.created
myapp.user.account.dlt

👉 Use .dlt as a suffix for failed messages.

Example Topic List

Topic	Producer	Consumer	Description
myapp.user.account.created	user-service	data-service	User created
myapp.user.account.updated	user-service	data-service	User updated
myapp.user.account.deleted	user-service	data-service	User deleted
myapp.user.account.dlt	(DLT handler)	data-service	Failed user events
myapp.report.file.created	report-service	data-service	File created
myapp.report.file.updated	report-service	-	File updated
myapp.report.file.deleted	report-service	data-service	File deleted
myapp.report.file.dlt	(DLT handler)	data-service	Failed report events
myapp.order.created	report-service	-	Order created

Partition Key

Use a consistent key across topics:

👉 userId

Why:

Ensures events for the same user go to the same partition
Preserves ordering

3. Debugging Topics & Messages

Using Kafka UI (Recommended)

Open:

http://<KAFKA_HOST_IP>:8989

You can:

View topics and partitions
Inspect messages in real time
Monitor consumer lag

Using CLI (Inside Container)

# List topics
docker exec -it app-kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9090 --list

# Read messages
docker exec -it app-kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9090 \
  --topic myapp.report.file.created \
  --from-beginning

# Produce test message
docker exec -it app-kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9090 \
  --topic myapp.report.file.created \
  --property "key.separator=:" \
  --property "parse.key=true"

4. Common Rules (From Real Projects)

Message Size

Keep payload small (IDs, references only)
Avoid sending large files directly
Use storage (S3, GCS, etc.) and pass the path

👉 Recommended: under 1MB

Idempotency

Always include:

idempotencyKey (UUID)

This prevents duplicate processing when messages are retried.

Error Handling Strategy

Scenario	Strategy
Producer failure	Log and continue (fire-and-forget)
Consumer failure	Retry → then send to DLT
DLT message	Alert + manual review
Wrong ACK usage	Avoid committing on failure (data loss risk)

What’s Next

Now that the infrastructure is ready, we can move on to implementation.

Using Kafka in Spring Boot → KafkaTemplate + @KafkaListener
Using Kafka in FastAPI → aiokafka + async producer

Final Thoughts

Setting up Kafka locally is easier than it looks, and having a clear topic design from the beginning will save you a lot of trouble later.

In real projects, the biggest problems usually come from:

Poor naming conventions
Inconsistent partition keys
Missing idempotency handling

If you get these basics right, Kafka becomes a very powerful and reliable backbone for your system.

#Kafka