Apache Kafka Tutorial — Concepts, Setup & Usage

Core Concepts

The foundational building blocks of every Kafka deployment.

Events (Records)

The fundamental unit of data — an immutable fact consisting of a key, value, timestamp, and optional headers. Once written to the log, events cannot be modified.

Topics

Named categories or "folders" where events are organized and stored. Producers write to topics and consumers read from them. Topics are the primary organizational unit.

Partitions

Each topic is split into ordered, append-only log segments distributed across the cluster. Partitions are the unit of parallelism — more partitions = more parallel consumers.

Brokers

Individual servers in a Kafka cluster. Each broker stores partitions, handles read/write requests, and replicates data to other brokers for fault tolerance.

Producers

Client applications that publish (write) events to specific Kafka topics. Producers choose which partition to write to based on the event key or a custom partitioner.

Consumers & Groups

Consumers subscribe to topics. A Consumer Group ensures each partition is read by exactly one consumer in the group — enabling parallel processing and load balancing.

Offsets

Unique, sequential IDs assigned to each event within a partition. Offsets act as bookmarks — consumers track their position and can replay events from any offset.

KRaft Mode

Kafka 4.0+ uses the KRaft consensus protocol for cluster metadata management, fully replacing ZooKeeper. This simplifies operations and reduces external dependencies.

Architecture

How data flows through a Kafka cluster from producer to consumer.

📤

1. Produce

Producers serialize events and send them to a specific topic + partition. If a key is provided, events with the same key always go to the same partition (key-based ordering).

💾

2. Store & Replicate

Brokers write events to disk as an append-only commit log. Data is replicated across N replicas (configurable) for fault tolerance. One replica is the leader; others are followers.

📥

3. Consume

Consumers pull events from partitions, tracking their position via offsets. Consumer Groups distribute partitions across members for parallel processing.

                    /* Data Flow Diagram */

                    Producer A ──→ [Topic: orders] ──→ Partition 0 ──→ Broker 1 (Leader) ──→ Broker 2 (Follower)

                    Producer B ──→ [Topic: orders] ──→ Partition 1 ──→ Broker 2 (Leader) ──→ Broker 3 (Follower)

                    Producer C ──→ [Topic: orders] ──→ Partition 2 ──→ Broker 3 (Leader) ──→ Broker 1 (Follower)

                    Consumer Group "processors":

                      Consumer 1 ←── Partition 0

                      Consumer 2 ←── Partition 1

                      Consumer 3 ←── Partition 2

Setup Guide

Get Kafka running locally in minutes using KRaft mode (no ZooKeeper required).

Option 1: Docker (Recommended)

The fastest way to start Kafka locally with a single command:

                        # Run Kafka in KRaft mode (no ZooKeeper)

                        docker run -d --name kafka \

                          -p 9092:9092 \

                          -e KAFKA_CFG_NODE_ID=1 \

                          -e KAFKA_CFG_PROCESS_ROLES=broker,controller \

                          -e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \

                          -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \

                          bitnami/kafka:latest

Option 2: Native Install (KRaft)

Download from kafka.apache.org and run with KRaft mode:

                        # 1. Generate cluster ID

                        KAFKA_CLUSTER_ID=$(bin/kafka-storage.sh random-uuid)

                        # 2. Format storage directories

                        bin/kafka-storage.sh format \

                          -t $KAFKA_CLUSTER_ID \

                          -c config/kraft/server.properties

                        # 3. Start Kafka

                        bin/kafka-server-start.sh config/kraft/server.properties

Usage Examples

Creating topics, producing messages, and consuming events with CLI and Python.

CLI Commands

                        # Create a topic with 3 partitions

                        bin/kafka-topics.sh --create \

                          --topic user-events \

                          --partitions 3 \

                          --replication-factor 1 \

                          --bootstrap-server localhost:9092

                        # List all topics

                        bin/kafka-topics.sh --list \

                          --bootstrap-server localhost:9092

                        # Produce messages (interactive)

                        bin/kafka-console-producer.sh \

                          --topic user-events \

                          --bootstrap-server localhost:9092

                        # Consume messages from beginning

                        bin/kafka-console-consumer.sh \

                          --topic user-events \

                          --from-beginning \

                          --bootstrap-server localhost:9092

Python (kafka-python)

                        # pip install kafka-python

                        from kafka import KafkaProducer, KafkaConsumer

                        # --- Producer ---

                        producer = KafkaProducer(

                          bootstrap_servers='localhost:9092',

                          value_serializer=lambda v: json.dumps(v).encode()

                        )

                        producer.send('user-events', {

                          'user_id': 42, 'action': 'click'

                        })

                        producer.flush()

                        # --- Consumer ---

                        consumer = KafkaConsumer(

                          'user-events',

                          bootstrap_servers='localhost:9092',

                          auto_offset_reset='earliest',

                          group_id='my-app-group'

                        )

                        for msg in consumer:

                          print(msg.value)

Best Practices

Production-grade recommendations for reliable Kafka deployments.

Partition Strategy

Choose partition count based on desired throughput. More partitions = more parallelism. Use key-based partitioning for ordering guarantees. Start with 6-12 partitions per topic.

Idempotent Producers

Enable enable.idempotence=true to prevent duplicate messages on retries. This ensures exactly-once semantics at the producer level without application-level deduplication.

Consumer Lag Monitoring

Monitor consumer lag (difference between latest offset and committed offset) using tools like Burrow or Kafka Exporter for Prometheus. Alert when lag exceeds thresholds.

Retention Policy

Configure retention.ms based on your use case. Use compacted topics for changelog patterns where you only need the latest value per key.

Schema Registry

Use Confluent Schema Registry or Apicurio to version and validate Avro/Protobuf schemas. Prevents breaking producer-consumer contracts.

Replication Factor

Always set replication factor ≥ 3 in production. Configure min.insync.replicas=2 with acks=all to prevent data loss.

Scaling & Production

Partition Scaling

More partitions = more parallelism. Target 1–4 MB/s per partition. For 100 MB/s throughput, use ~50–100 partitions. Partitions can only be increased, never decreased — plan capacity upfront.

Broker Horizontal Scaling

Add brokers to the cluster and use kafka-reassign-partitions.sh to rebalance. KRaft mode supports dynamic broker addition without ZooKeeper coordination. Monitor under-replicated partitions during rebalance.

Consumer Group Scaling

Scale consumers up to # partitions — each partition is assigned to exactly one consumer in the group. Beyond that, use multiple consumer groups for fan-out patterns or materialized views.

Tiered Storage

Offload cold data to object storage (S3/GCS) with Tiered Storage (KIP-405, Confluent). Keeps broker disk usage bounded while retaining weeks/months of data. Hot data stays on local NVMe for low-latency reads.

Multi-Datacenter

Use MirrorMaker 2 for cross-datacenter replication. Supports active-active topologies with topic renaming and offset translation. For cloud deployments, Cluster Linking (Confluent) provides byte-level replication with lower latency.

Throughput Benchmarks

A 3-broker cluster on modern hardware can handle 2+ GB/s write throughput. Key levers: batch size (batch.size=1MB), compression (lz4), linger.ms=5, and NVMe storage. Profile with kafka-producer-perf-test.sh.

Frequently Asked Questions

Kafka vs RabbitMQ — when to use which? →

Use Kafka for high-throughput event streaming, log aggregation, and replay scenarios where you need durable message retention. Use RabbitMQ for traditional message queuing with complex routing patterns (fanout, topic exchanges) and when messages should be consumed once and discarded.

Does Kafka guarantee exactly-once delivery? →

Yes — Kafka supports exactly-once semantics (EOS) when using idempotent producers + transactional APIs. Enable enable.idempotence=true and transactional.id on the producer, and isolation.level=read_committed on consumers.

How long does Kafka retain messages? →

By default, Kafka retains messages for 7 days (168 hours). This is configured per-topic via retention.ms. You can set it to -1 for infinite retention, or use compacted topics to keep only the latest value per key.

Can I change the number of partitions after creation? →

You can increase partitions, but you cannot decrease them. Increasing partitions breaks key-based ordering guarantees for existing keys. Plan your partition count carefully at topic creation time.

What happened to ZooKeeper? →

Apache Kafka 4.0 (released 2025) fully removed ZooKeeper support. All new deployments use KRaft mode, which manages cluster metadata internally using the Raft consensus protocol. This simplifies deployment and reduces operational overhead.