The Question
Design

Distributed Message Queue

Design a distributed message queue system similar to Kafka or RabbitMQ. The system should provide reliable and ordered message delivery, support multiple producers and consumer groups, guarantee at-least-once delivery, and scale to handle millions of messages per second with low latency.
Append-Only Log
Zero-Copy
Partitioning
Replication
etcd/ZooKeeper
Questions & Insights

Thinking Process

To design a production-grade Distributed Message Queue (DMQ), focus on the "Log-Structured" paradigm. The core challenge is balancing high-throughput writes with reliable, ordered reads.
How do we handle massive scale without a single-point bottleneck? Partitioning (Sharding). We split a "Topic" into multiple "Partitions" distributed across different brokers to parallelize I/O.
How do we guarantee durability and availability? WAL (Write-Ahead Log) and Replication. Every message is appended to a disk-backed log and replicated to "Follower" brokers before acknowledgment.
How do we manage consumer progress efficiently? Offset Management. Instead of tracking individual message acknowledgments (which is expensive), we track a single "high-water mark" or offset per consumer group per partition.
How do we achieve high performance on commodity hardware? Zero-copy optimization and Sequential I/O. Use sendfile to bypass user-space buffers and treat the disk as a sequential append-only stream.

Bonus Points

Zero-Copy I/O: Using mmap or sendfile to transfer data directly from the OS Page Cache to the Network Socket, bypassing the JVM/Application heap to reduce CPU cycles and memory bandwidth.
Tiered Storage: Decoupling compute from storage by offloading older log segments to S3/GCS, allowing for "infinite" retention without scaling the expensive broker local SSDs.
ISR (In-Sync Replicas) Model: Implementing a dynamic quorum (Kafka-style) where the "Leader" only waits for replicas that are "caught up," balancing strict consistency with system availability during transient network partitions.
Batching & Compression: Implementing end-to-end compression (Producer compresses, Broker stores raw, Consumer decompresses) to minimize IOPS and network saturation.
Design Breakdown

Functional Requirements

Producers can send messages to specific "Topics."
Consumers can subscribe to Topics and receive messages.
Durability: Messages must be persisted to disk and survive broker restarts.
Ordering: Guarantee per-partition message ordering.
Retention: Messages remain available for a configurable period (e.g., 7 days) or size limit.

Non-Functional Requirements

High Throughput: Support millions of messages per second.
Low Latency: Sub-10ms end-to-end latency for P99.
Scalability: Horizontally scale by adding more brokers.
Fault Tolerance: No data loss on single node failure (N+1 redundancy).

Estimation

Throughput: 1,000,000 messages/sec.
Message Size: 1 KB average.
Bandwidth: 1,000,000 * 1 KB = 1 GB/s (Ingress). With 3x replication = 3 GB/s total internal traffic.
Storage: 1 GB/s * 86,400s (1 day) ≈ 86 TB/day.
Retention (7 days): ~600 TB total.
Infrastructure: Assuming 100 MB/s per node disk I/O, we need ~30 nodes minimum for storage/throughput.

Blueprint

Concise Summary: A partitioned, append-only distributed log system where brokers handle storage/retrieval, and a coordination service manages cluster state.
Major Components:
Producer: Client-side library that handles batching and routing messages to specific partitions via hashing or round-robin.
Broker: The core engine responsible for appending messages to local disk logs and serving read requests from consumers.
Metadata Store (Etcd/Zookeeper): Stores the "Source of Truth" regarding cluster topology, partition leadership, and consumer offsets.
Consumer Group Coordinator: Logic within the broker to manage partition assignment among a group of consumers.
Simplicity Audit: This architecture avoids complex "push" logic by using a "pull" model, shifting the burden of flow control to consumers and simplifying broker state management.

High Level Architecture

Sub-system Deep Dive

Service

Topology & Scaling: Stateless Producers and Consumers. Brokers are stateful but share no disk; scaling is achieved by rebalancing partitions across new nodes.
API Spec:
Produce(topic, partition, message_bytes): Binary protocol over TCP for speed.
Fetch(topic, partition, offset): Pull-based request to retrieve a batch of messages.
Heartbeat(): For consumers to maintain group membership.

Storage

Data Model: Messages are stored as a sequence of bytes. On disk, a Partition is a directory containing "Segment Files."
Index File: Maps message offsets to physical byte positions in the segment file.
Log File: The raw message data appended sequentially.
Database Logic: No RDBMS. Uses OS Page Cache extensively. Writes are fsync()ed based on policy (e.g., every N ms or every N messages) to balance performance and safety.

Messaging

Topic Structure: Divided into N partitions. Each partition has 1 Leader and M Followers.
Delivery Guarantees:
At-least-once: Default. Producer retries on failure; Consumer commits offset after processing.
Exactly-once: Supported via Idempotent Producers (Sequence IDs) and Atomic Transactions across partitions.
Consumers: Use a "Pull" model to prevent overwhelming slow consumers (backpressure is inherent).
Wrap Up

Advanced Topics

Trade-offs:
Latency vs. Durability: Waiting for all replicas to ACK (all-in-sync) increases latency but ensures zero data loss.
Pull vs. Push: Pull-based consumers allow for easier batching and flow control but may introduce a slight delay if the polling interval is high.
Bottlenecks:
Disk I/O: High-volume small writes. Solved by batching at the producer level.
Network Bandwidth: Specifically for replication traffic. Solved by using 10Gbps+ NICs and intra-AZ traffic optimization.
Failure Handling:
Broker Failure: Metadata store detects heartbeat loss; triggers leader re-election for affected partitions.
Metadata Failure: Use a highly available consensus-based store (Etcd/Zookeeper) with 3 or 5 nodes to survive minority failure.
Alternatives & Optimization:
Alternative: Use Pulsar instead of a Kafka-style design if "work queue" features (individual message ACKs) are required alongside streaming.
Optimization: Implement Compacted Topics where only the latest value for a specific key is kept, saving storage for state-tracking use cases.