The Question
Design

Scalable Social Media Feed Engine (Twitter-style)

Design a high-scale social media platform similar to Twitter, focusing on the core challenges of tweet propagation, low-latency feed generation, and handling extreme fan-out for high-profile accounts (celebrities). The system must support 300 million daily active users, maintain sub-200ms timeline latency, and handle peak loads of 100,000 tweets per second. Discuss your choice of database for the social graph vs. tweet storage, your strategy for pre-computing feeds, and how you ensure system reliability during traffic spikes.
Cassandra
Redis
Kafka
PostgreSQL
CDN
Snowflake ID
Zookeeper
Microservices
Questions & Insights

Clarifying Questions

Scale and Traffic: What is the expected Daily Active User (DAU) count, and what is the read/write ratio? (Assumption: 300M DAU, 500M tweets/day, ~100:1 read-to-write ratio).
Content Types: Does the MVP support media (images/video) or just text? (Assumption: Text and image URLs; media storage is handled by an object store like S3).
Fan-out Constraints: How should we handle "celebrity" accounts with millions of followers? (Assumption: A hybrid push/pull model is required to prevent "thundering herd" issues in the fan-out service).
Timeline Ordering: Is the timeline strictly chronological or algorithmic? (Assumption: Reverse-chronological for the MVP).
Search/Discovery: Is full-text search required for the MVP? (Assumption: No, focus on core feed and following mechanics).

Thinking Process

The Fan-out Problem: The core bottleneck is propagating a single tweet to millions of follower timelines.
Progressive Questions:
How do we store tweets and user relationships for high-scale reads?
How do we ensure low-latency timeline delivery (Pre-computation vs. On-the-fly)?
How do we handle "Hot Keys" (celebrities) whose tweets break the standard push-based fan-out?
How do we ensure high availability when the fan-out service lags?

Bonus Points

Snowflake ID Generation: Using a distributed unique ID generator (like Twitter's Snowflake) to ensure tweet IDs are roughly time-ordered and unique across shards without a central bottleneck.
Hybrid Fan-out Model: Implementing a "push" model for regular users and a "pull" (on-demand merge) model for celebrities to prevent Redis memory exhaustion and write-latency spikes.
Secondary Indexing with Bloom Filters: Using Bloom filters at the Edge/API layer to quickly check if a user has already "liked" or "seen" a tweet, reducing unnecessary DB hits.
Geo-sharding & Read Replicas: Placing timeline caches (Redis) close to the user's geographic region to minimize RTT (Round Trip Time).
Design Breakdown

Functional Requirements

Core Use Cases:
User can post a tweet.
User can follow/unfollow other users.
User can view a "Home Timeline" (tweets from people they follow).
User can view a "User Timeline" (their own tweets).
Scope Control:
In-scope: Tweeting, Following, Feed generation, Celebrity handling.
Out-of-scope: Search, Direct Messages, Trends, Analytics, Ad-engine.

Non-Functional Requirements

Scale: Support 300M DAU and 100k+ peak write QPS.
Latency: Timeline loading should be < 200ms (P99).
Availability & Reliability: 99.99% availability; tweets must not be lost once acknowledged.
Consistency: Eventual consistency is acceptable for timelines and follower counts.
Fault Tolerance: System must survive the failure of a single AZ or cache cluster.

Estimation

Traffic Estimation:
DAU: 300M.
Tweets/day: 500M.
Avg Write QPS: 500M / 86400 ≈ 6,000.
Peak Write QPS: 2x - 3x avg ≈ 15,000.
Read QPS (Timeline): 300M 10 visits/day / 86400 20 tweets per view ≈ 700,000 QPS.
Storage Estimation:
Avg Tweet: 280 bytes + metadata ≈ 1KB.
Daily Storage: 500M * 1KB = 500GB/day.
5-Year Storage: 500GB 365 5 ≈ 900TB.
Bandwidth Estimation:
Ingress: 6,000 * 1KB ≈ 6MB/s.
Egress: 700,000 * 1KB ≈ 700MB/s.

Blueprint

Concise Summary: A microservices-based architecture using a hybrid fan-out strategy. Tweets are persisted in a wide-column store, while active timelines are pre-computed and cached in-memory for low-latency retrieval.
Major Components:
Tweet Service: Handles incoming writes and persists tweets to the primary database.
Follow Service: Manages the social graph (who follows whom).
Fan-out Worker: Asynchronously pushes tweet IDs to the Redis caches of followers.
Timeline Service: Aggregates cached tweet IDs and hydrates them with content for the user.
Redis Timeline Cache: Stores the list of tweet IDs for each user's home feed.
Simplicity Audit: This design prioritizes read latency by pre-computing feeds, which is the most common user action. It uses asynchronous processing to keep the write path fast.
Architecture Decision Rationale:
Why this architecture?: Twitter is fundamentally a "read-heavy" system where users expect instant feed updates. The hybrid push/pull model balances the load between write-time (fan-out) and read-time (merging celebrity tweets).
Functional Satisfaction: Covers posting, following, and feed retrieval via distinct services.
Non-functional Satisfaction: Scalable via horizontal sharding of Redis and NoSQL databases; highly available through async fan-out.

High Level Architecture

Sub-system Deep Dive

Edge (Optional)

Content Delivery & Traffic Routing:
CDN: Used for serving static assets (JS/CSS) and media content (images/videos) via geographic edge nodes.
DNS: Latency-based routing to the nearest AWS/GCP region.
Security & Perimeter:
API Gateway: Handles JWT-based Authentication, Rate Limiting (100 tweets/min per user), and SSL termination.

Service

Topology & Scaling: Stateless microservices deployed across multiple Availability Zones (AZs). Auto-scaling based on CPU and Request Count.
API Schema Design:
POST /v1/tweets: { text: string, media_ids: [] }. Returns tweet_id.
GET /v1/timeline/home: Returns list of Tweet objects with pagination (cursor-based).
POST /v1/users/{id}/follow: Idempotent follow action.
Resilience & Reliability:
Retries: Fan-out workers use exponential backoff for Redis writes.
Circuit Breakers: Timeline service fails gracefully (returns empty or cached-only feed) if the hydrated Tweet DB is slow.

Storage

Access Pattern:
Tweets: High volume writes, read by ID or UserID.
Follows: Read-heavy social graph queries.
Database Table Design:
Tweet Table (Cassandra): Partitioned by user_id. tweet_id (Snowflake) as clustering key.
Follow Table (PostgreSQL): follower_id, followee_id, created_at. Indexed on both IDs.
Technical Selection:
Tweet DB: Cassandra/ScyllaDB for high-throughput, horizontally scalable writes and efficient time-series queries.
Graph DB: For MVP, PostgreSQL is sufficient. At 10x scale, move to Neo4j or AWS Neptune.
Distribution Logic:
Sharding Tweet DB by user_id to keep a user's tweets on the same physical node for fast profile loads.

Cache

Purpose & Justification: Redis stores the "Home Timeline" (list of tweet IDs) for active users (active in last 30 days) to avoid expensive JOINs/Aggregations on read.
Key-Value Schema:
Key: TL:{user_id}, Value: List<tweet_id>.
Max size: 1000 items per timeline.
Technical Selection: Redis Cluster with LRU eviction.
Failure Handling: If Redis is down, the Timeline Service performs a "Pull" query directly from Cassandra (slower, but functional).

Messaging

Purpose & Decoupling: Kafka decouples the Tweet Service from the Fan-out process.
Event Schema: { "tweet_id": 123, "author_id": 456, "timestamp": 789 }.
Throughput & Partitioning: Partitioned by author_id to ensure ordering of tweets from the same user.
Technical Selection: Kafka for high throughput and durability.

Data Processing

Processing Model: Stream processing via Fan-out Workers.
Processing Logic:
Fetch follower_ids from Follow DB.
For each follower, update their Redis Timeline list.
Celebrity Check: If follower_count > 50,000, skip push; mark tweet for "Pull" delivery.
Technical Selection: Go or Java-based workers for high-concurrency I/O.

Infrastructure (Optional)

Observability: Prometheus for metrics (Latency, Fan-out lag), Jaeger for tracing request flow across services.
Distributed Coordination: Snowflake ID generation requires ZooKeeper or Etcd for coordination of worker IDs.
Wrap Up

Advanced Topics

Trade-offs: We choose Availability over Consistency (AP). A follower might see a tweet a few seconds late (eventual consistency), which is acceptable for social media.
The Celebrity Problem:
Push: For normal users (Author -> 1,000 Followers), we write to 1,000 Redis lists.
Pull: For celebrities (Author -> 50M Followers), we don't push. Instead, when a follower loads their feed, the Timeline Service pulls the celebrity's recent tweets and merges them with the cached push-feed.
Storage Optimization: Timelines in Redis are capped at 1000 entries. Older tweets are retrieved from Cassandra via "Load More".
Security: PII (email/phone) in User DB must be encrypted at rest. Rate limiting prevents API scraping.