DowngradedOur downstream service providers are currently experiencing outages, and our engineering team is actively working on a resolution. Some services—including the Solver, Partner, and Tools—are temporarily degraded with higher latency and lower bandwidth. Rest assured, Intervipedia, Solutions, and the Question Bank features are not impacted and remain fully operational.DowngradedOur downstream service providers are currently experiencing outages, and our engineering team is actively working on a resolution. Some services—including the Solver, Partner, and Tools—are temporarily degraded with higher latency and lower bandwidth. Rest assured, Intervipedia, Solutions, and the Question Bank features are not impacted and remain fully operational.DowngradedOur downstream service providers are currently experiencing outages, and our engineering team is actively working on a resolution. Some services—including the Solver, Partner, and Tools—are temporarily degraded with higher latency and lower bandwidth. Rest assured, Intervipedia, Solutions, and the Question Bank features are not impacted and remain fully operational.DowngradedOur downstream service providers are currently experiencing outages, and our engineering team is actively working on a resolution. Some services—including the Solver, Partner, and Tools—are temporarily degraded with higher latency and lower bandwidth. Rest assured, Intervipedia, Solutions, and the Question Bank features are not impacted and remain fully operational.
DowngradedOur downstream service providers are currently experiencing outages, and our engineering team is actively working on a resolution. Some services—including the Solver, Partner, and Tools—are temporarily degraded with higher latency and lower bandwidth. Rest assured, Intervipedia, Solutions, and the Question Bank features are not impacted and remain fully operational.DowngradedOur downstream service providers are currently experiencing outages, and our engineering team is actively working on a resolution. Some services—including the Solver, Partner, and Tools—are temporarily degraded with higher latency and lower bandwidth. Rest assured, Intervipedia, Solutions, and the Question Bank features are not impacted and remain fully operational.DowngradedOur downstream service providers are currently experiencing outages, and our engineering team is actively working on a resolution. Some services—including the Solver, Partner, and Tools—are temporarily degraded with higher latency and lower bandwidth. Rest assured, Intervipedia, Solutions, and the Question Bank features are not impacted and remain fully operational.DowngradedOur downstream service providers are currently experiencing outages, and our engineering team is actively working on a resolution. Some services—including the Solver, Partner, and Tools—are temporarily degraded with higher latency and lower bandwidth. Rest assured, Intervipedia, Solutions, and the Question Bank features are not impacted and remain fully operational.
The Question
Design

Scalable Social Media Feed System

Design a high-scale social media platform similar to Twitter. The system must handle millions of concurrent users posting updates and consuming a real-time chronological home feed. Key challenges include managing a massive social graph, handling 'celebrity' accounts with millions of followers without overwhelming the system (the fan-out problem), and ensuring sub-200ms timeline loading speeds. Discuss your choice of data storage, caching strategies, and how you would balance push vs. pull models for feed generation.
Cassandra
Redis
Kafka
PostgreSQL
CDN
Kubernetes
gRPC
ZSET
Questions & Insights

Clarifying Questions

What is the scale of the system? (Assumption: 200M DAU, 100M tweets per day, average 500 follows per user).
What are the core features for the MVP? (Assumption: Posting tweets, following/unfollowing users, viewing a Home Timeline, and viewing a User Profile Timeline).
Is media support (images/videos) required? (Assumption: Yes, but the focus is on the feed architecture; media is handled via Object Storage/CDN).
What are the latency requirements for the timeline? (Assumption: Home timeline should load in < 200ms).
How do we handle "Celebrities" or high-fanout users? (Assumption: We need a specific strategy for users with millions of followers to avoid "thundering herd" issues).

Thinking Process

The Fan-out Bottleneck: How do we efficiently deliver a tweet to 10M followers without crushing the database?
Read vs. Write Heavy: Twitter is read-heavy (approx. 100:1 ratio). We must optimize for fast reads using pre-computation.
Hybrid Timeline Model: Use "Push" (Fan-out) for regular users to keep their timelines ready in memory, and "Pull" (On-read) for celebrities to avoid massive write amplification.
Data Sharding: How do we partition the tweet storage and social graph to scale horizontally?

Bonus Points

Cell-Based Architecture: Deploying the system in isolated "cells" to limit the blast radius of failures and facilitate easier regional expansion.
Cache-Aside with Lease/Promise: Using a "Lease" mechanism in Redis to prevent cache stampedes when a celebrity tweet finally expires or is updated.
Social Graph Compression: Utilizing specialized graph databases or adjacency list compression to manage trillions of follow edges efficiently.
Selective Fan-out: Using real-time analytics to only fan out to "active" followers (users logged in within the last 30 days) to save IOPS.
Design Breakdown

Functional Requirements

Core Use Cases:
User can post a tweet (text, media metadata).
User can follow/unfollow other users.
User can view a "Home Timeline" (posts from people they follow, sorted by time).
User can view a "User Timeline" (posts from a specific user).
Scope Control:
In-scope: Tweeting, Following, Timeline generation, Basic Media metadata.
Out-of-scope: Search, Direct Messages, Trending Topics, Analytics/Ads, Retweets/Quotes (simplified for MVP).

Non-Functional Requirements

Scale: Support 200M DAU and 100k+ QPS for writes, 10M+ QPS for reads.
Latency: Home timeline delivery under 200ms (P99).
Availability & Reliability: 99.99% availability (Highly Available); Tweets must not be lost once acknowledged.
Consistency: Eventual consistency is acceptable for timelines (it's okay if a tweet takes 5 seconds to appear to all followers).
Fault Tolerance: No single point of failure; graceful degradation if the fan-out service lags.

Estimation

Traffic:
Tweets: 100M/day \approx 1,150 tweets/sec (Avg). Peak: 5k - 10k/sec.
Timeline Reads: 200M users * 5 visits/day \approx 1B views/day \approx 12k/sec (Avg). Peak: 100k+ QPS.
Storage:
Tweet: 280 chars + metadata \approx 1KB.
100M tweets/day * 1KB \approx 100GB/day. 36.5 TB/year.
Bandwidth:
Ingress: 10k tweets/sec * 1KB \approx 10MB/s (Text only).
Egress: 100k reads/sec 20 tweets/view 1KB \approx 2GB/s.

Blueprint

Concise Summary: A hybrid feed system that pre-computes timelines for normal users via an asynchronous fan-out worker and dynamically merges celebrity tweets at read-time.
Major Components:
Tweet Service: Handles incoming writes and persists tweets to the primary store.
Social Graph Service: Manages follow/unfollow relationships.
Fan-out Worker: Asynchronously pushes new tweet IDs into followers' Redis-based timeline caches.
Timeline Service: Aggregates and serves the final feed to the user.
Simplicity Audit: This design avoids complex Machine Learning ranking for the MVP, using strict chronological order, which significantly simplifies storage and caching.
Architecture Decision Rationale:
Why this architecture?: Pure "Pull" models (SQL JOIN on followings) are too slow for millions of followers. Pure "Push" models (writing to every follower's list) break when a celebrity tweets. The Hybrid approach balances these constraints.
Functional Satisfaction: Covers the full lifecycle from tweet creation to timeline consumption.
Non-functional Satisfaction: High availability via stateless services and horizontal scaling via sharded NoSQL and Redis clusters.

High Level Architecture

Sub-system Deep Dive

Edge (Optional)

Content Delivery & Traffic Routing: CDN (e.g., Cloudflare) used to cache static assets and media. Geographic DNS routes users to the nearest regional data center.
Security & Perimeter: API Gateway handles JWT-based AuthN/AuthZ, rate-limiting (to prevent bot spamming), and TLS termination.

Service

Topology & Scaling: Stateless microservices deployed on Kubernetes. Horizontal Pod Autoscaler (HPA) triggers on CPU/QPS.
API Schema Design:
POST /v1/tweets: Creates a tweet. Returns tweet_id.
GET /v1/timeline/home: Returns list of tweet objects. Supports cursor-based pagination (since_id).
POST /v1/follow/{userId}: Idempotent follow action.
Resilience: Fan-out is asynchronous; if it fails, it retries from the message queue. Timeline service can fall back to a "Pull" model from the DB if Redis is cold.

Storage

Access Pattern: Tweet store is write-once, read-many. Graph store requires high-performance adjacency lookups.
Database Table Design:
Tweet Store (NoSQL - e.g., Cassandra): Partitioned by user_id for User Timelines, or tweet_id for global lookups.
Graph Store (RDBMS - e.g., PostgreSQL): Table follows (follower_id, followee_id). Indexed on both columns.
Technical Selection:
Cassandra: High write throughput and easy horizontal scaling for tweet data.
PostgreSQL: Strong consistency for social graph (following/unfollowing) to ensure users don't see "ghost" follows.

Cache

Purpose: Reducing latency for timeline generation.
Key-Value Schema:
Key: TL:{user_id}, Value: List of tweet_ids (Redis LIST or ZSET).
TTL: 72 hours (Inactive users' timelines are evicted).
Failure Handling: If Redis fails, the Timeline service queries the Graph Store for "followees" and then queries the Tweet Store (Pull model).

Messaging

Purpose: Decoupling tweet creation from the expensive fan-out process.
Event Schema: TweetCreatedEvent { tweet_id, author_id, timestamp, is_celebrity }.
Technical Selection: Kafka. High throughput and retention allow for re-processing if fan-out logic changes or fails.

Data Processing

Processing Model: Stream processing (the Fan-out Workers).
Logic: For every TweetCreatedEvent, fetch follower_ids from Graph Service. For each follower_id, LPUSH the tweet_id into their Redis Timeline.
Celebrity Logic: If is_celebrity is true, skip the fan-out. The Timeline service will inject these tweets at read-time.

Infrastructure (Optional)

Observability: Prometheus for metrics (QPS, Error rates), Jaeger for tracing tweet lifecycle from API to Fan-out.
Wrap Up

Advanced Topics

The "Celebrity" Trade-off:
Problem: Fans of Lady Gaga (90M+ followers) would cause a massive write spike if we fanned out her tweets to every Redis list.
Solution: Celebrity tweets are NOT fanned out. When a follower of Lady Gaga loads their timeline, the Timeline Service pulls the last few tweet_ids from Lady Gaga's User Timeline and merges them with the Redis-based pre-computed feed.
Bottleneck Analysis:
Hot Partitions: Large user followings. Addressed via the hybrid model.
Redis Memory: Storing 200M timelines is expensive. Optimization: Only store timelines for users active in the last 30 days.
Consistency vs. Availability: Under PACELC, we choose Availability (AP) for the feed. If a user doesn't see a friend's tweet for 2 seconds, it's fine.
Security: PII (email, IP) is encrypted at rest. Following relationships are public by default unless the account is private (handled via AuthZ check in the Graph Service).