The Question
Design

End-to-End Encrypted Messaging App

Design a real-time messaging application similar to WhatsApp with end-to-end encryption. The system should support one-on-one and group chats, media sharing, offline message delivery via push notifications, and read receipts at massive global scale.
WebSocket
Redis
Cassandra
Signal Protocol
Message Broker
Questions & Insights

Thinking Process

To design a system like WhatsApp, the core bottleneck is managing persistent connections for millions of concurrent users and ensuring message delivery in a secure, encrypted manner.
How do we handle real-time delivery? Use persistent WebSockets or MQTT for bi-directional communication to minimize latency compared to HTTP polling.
How do we manage user "Presence"? Use a distributed Key-Value store (Redis) to track which WebSocket server a specific user is connected to.
How do we ensure End-to-End Encryption (E2EE)? Implement the Signal Protocol. The server acts as a "dumb pipe," storing only public keys and encrypted blobs, never the plaintext or private keys.
How do we handle offline users? Messages are persisted in a high-write database (Cassandra) and delivered via push notifications (FCM/APNS) when the user reconnects.

Bonus Points

Signal Protocol (X3DH & Double Ratchet): Mentioning "Perfect Forward Secrecy" by rotating keys for every message ensures that even if one key is compromised, past and future messages remain secure.
Key Transparency: Implementing a verifiable log (like a Merkle Tree) for public keys to prevent Man-in-the-Middle (MITM) attacks by the service provider itself.
Binary Protocols: Using Protocol Buffers (Protobuf) instead of JSON over WebSockets to reduce payload size, saving battery life and bandwidth on mobile devices.
Write-Path Optimization: Using an LSM-tree based storage (Cassandra/ScyllaDB) to handle the massive write-heavy workload of trillions of messages.
Design Breakdown

Functional Requirements

One-to-one real-time text messaging.
End-to-end encryption (E2EE).
Message status (Sent, Delivered, Read).
User presence (Online/Offline/Last Seen).
Media support (Image/Video) - MVP focuses on metadata and secure upload/download flow.

Non-Functional Requirements

Low Latency: Near-instant delivery (<200ms).
High Availability: System must be 99.99% available.
Scalability: Support 100M+ Daily Active Users (DAU).
Security: Zero-knowledge architecture (Server cannot read messages).

Estimation

DAU: 100 Million.
Messages per day: 50 per user = 5 Billion messages/day.
Average Message Size: 100 bytes (text + metadata).
Total Write Throughput: 5B / 86400 seconds ≈ 60,000 Messages Per Second (MPS).
Peak Throughput: 3x average ≈ 180,000 MPS.
Storage (1 Year): 5B 100 bytes 365 days ≈ 180 TB. (Requires distributed NoSQL).

Blueprint

Concise Summary: A WebSocket-based architecture using the Signal Protocol for encryption, with Redis for session tracking and Cassandra for horizontally scalable message persistence.
Major Components:
WebSocket Service: Maintains persistent stateful connections for real-time message relay.
Presence Service: Tracks user heartbeats and connection status using Redis.
Message Broker: Decouples message ingestion from delivery, ensuring reliability during traffic spikes.
Key Transparency Store: A relational database storing users' public keys (Identity keys, Pre-keys) for E2EE setup.
Simplicity Audit: This design avoids complex microservices for "Status" or "Stories" to focus on the core reliability of 1:1 messaging. It uses a single NoSQL cluster for messages to simplify the data layer.
Architecture Decision Rationale:
Why this architecture is the best?: WebSockets are essential for low-latency bi-directional communication. Cassandra is chosen because it is optimized for high-volume writes and scales linearly.
Functional Requirement Satisfaction: E2EE is satisfied by the Key Store; delivery receipts are handled by the bi-directional WS protocol.
Non-functional Requirement Satisfaction: Scalability is achieved by partitioning the Message Store by conversation_id and the Presence Store by user_id.

High Level Architecture

Sub-system Deep Dive

Service

Topology & Scaling: WebSocket nodes are stateful. We use a Connection Manager to map user_id -> server_ip in Redis. Scaling is achieved by adding more nodes and updating the load balancer (using consistent hashing or sticky sessions).
API Spec:
POST /v1/keys: Upload initial bundle of E2EE pre-keys.
GET /v1/keys/{user_id}: Fetch the public key bundle for a recipient.
WS /connect: Establish persistent connection with JWT auth.
Message Format (Binary): [Header: MessageID, RecipientID][Body: EncryptedBlob][Hmac: Signature].

Storage

Data Model:
Message Store (Cassandra): Partition Key: conversation_id, Clustering Key: timestamp. This allows extremely fast reads for recent chat history.
Key Store (Postgres): Stores user_id, public_identity_key, and a pool of signed_pre_keys.
Database Logic: We use "Last-Write-Wins" for message status updates. Messages are deleted from the server once a "Read" receipt is acknowledged by the sender (optional, for privacy).

Cache

Data Structures: Redis Hash presence:{user_id} stores status (online/offline), last_seen (timestamp), and server_id (the WS node the user is currently on).
TTLs: Presence heartbeats are sent every 30 seconds; Redis keys have a 60-second TTL to handle abrupt disconnections.
Eviction: Standard LRU, though the dataset is small enough to fit in RAM for millions of users.

Messaging

Topic Structure: One topic message_delivery.
Delivery Guarantees: At-least-once delivery. Each message has a unique client-generated UUID to handle de-duplication at the consumer level.
Consumers: The Message Consumer reads from the broker, persists the message to Cassandra, and checks the Presence Cache. If the user is online, it routes the message to the specific WebSocket node. If offline, it triggers the Push Notification Service.
Wrap Up

Advanced Topics

Monitoring:
Metrics: WebSocket connection count, Message E2E latency, Redis heartbeat miss rate, Push notification delivery success rate.
Tools: Prometheus for metrics, Grafana for visualization.
Trade-offs:
Consistency vs Availability: We choose Availability (AP) for message delivery. If a node fails, messages might be retried or delivered slightly out of order, but the system remains functional.
Bottlenecks: The WebSocket server's memory is the limit (each connection takes ~10-50KB). 1M connections require ~50GB RAM.
Failure Handling:
WS Node Crash: Client detects socket close, reconnects to a different node via LB, and fetches missed messages from the Message Store.
Alternatives & Optimization:
Optimization: Use Message Sequence Numbers per conversation to detect gaps in message history locally on the device.
Alternative: Use SQLite on the mobile device to store message history locally, only using the server as a relay and temporary buffer.