Message Queue (Kafka)
High-throughput distributed append-only log queue decoupling service communications and managing asynchronous workloads.
1. System Requirements
- Publish and subscribe to message topic streams asynchronously.
- Persist messages to disk to enable replayability and fault tolerance.
- Distribute message processing across consumers using partition keys.
- High write throughput: millions of messages per second.
- Guaranteed order per partition.
2. Scale & Resource Estimates
Throughput: Scale to gigabytes of writes per second. Clusters handle millions of concurrent messages.
Storage & RAM: Disk-bound append-only commit logs. Rotated based on time (e.g. 7 days retention) or log size limits.
3. System API Endpoints
// Producer publish API
Producer.send(topic="payment-events", key="usr-4412", value={
"order_id": "ord-883",
"amount": 250.00,
"timestamp": 1783708910
})4. Database Schema Design
Topic Commit Log (Partition offsets):
Offset 1045: { "order_id": "ord-881", "status": "pending" }
Offset 1046: { "order_id": "ord-882", "status": "completed" }5. Interactive High-Level Architecture
6. Scalability Deep Dive & Bottlenecks
Kafka achieves high write speeds by treating partitions as append-only commit logs on disk. Writes are sequential, bypassing random-access seek delays on HDD/SSDs. Consumers read messages from specific offsets, allowing they to pause, resume, and replay streams. Messages with the same partition key are guaranteed to land on the same broker node, maintaining chronological execution orders.
7. Production Scaling Strategies
Partitioning: Splitting a single topic across multiple broker servers to scale writes and reads horizontally.
Consumer Groups: Grouping worker nodes to read partitions in parallel. Partition count sets the maximum consumer limit.
Zero-Copy reads: Using kernel-level page caching and sendfile system calls to bypass user-space memory buffers on reads.
8. Runnable Production Starter Code
Scaffold