The Question
Design

Social Media Microblogging Platform

Design a social media platform similar to Twitter. The system should support posting short messages, following other users, delivering personalized real-time feeds, handling viral content fan-out, and serving hundreds of millions of daily active users at low latency.
Redis
Kafka
PostgreSQL
Fan-Out
Microservices
Questions & Insights

Thinking Process

The core challenge of Twitter is the Fan-out Problem: how to efficiently deliver a single tweet to millions of followers' feeds in near real-time.
The Read/Write Asymmetry: Twitter is read-heavy (100:1 or 1000:1). We must optimize for the "Read" path (Timeline generation) by pre-computing feeds during the "Write" path.
The Celebrity (Hot Key) Problem: A standard "Push" model breaks when a celebrity with 100M followers tweets. We need a hybrid approach or a specialized bypass for high-degree nodes.
Data Modeling for Graph: How do we store "Follow" relationships to ensure sub-millisecond lookups?
Logical Flow: How does a tweet go from a user's phone to the Redis-based pre-computed timelines of 5,000 followers?

Bonus Points

Hybrid Push/Pull Model: For MVP, we use "Push" (Fan-out on write). However, for "Celebrities" (e.g., >100k followers), we switch to "Pull" (Load their tweets on-demand) to prevent the "Thundering Herd" effect on the Fan-out workers.
Geo-Sharding and Edge Caching: Using CDNs for media is standard, but sharding the timeline cache by user geography reduces cross-region latency for global feeds.
Timeline Triage: Implementing a "Visible Timeline" vs. "Archived Timeline." We only pre-compute Redis feeds for active users (logged in within the last 30 days) to save massive memory costs.
Idempotent Writes: Using Snowflake IDs for tweets to ensure that retries from the client don't result in duplicate posts.
Design Breakdown

Functional Requirements

Users can post tweets (text-focused for MVP).
Users can follow/unfollow other users.
Users can view a Home Timeline (tweets from people they follow).
Users can view a User Timeline (their own tweets).

Non-Functional Requirements

High Availability: The system must be available 99.99% of the time (Social media over consistency).
Low Latency: Home timeline generation must be < 200ms.
High Scalability: Support 300M Daily Active Users (DAU).
Eventual Consistency: It is acceptable if a tweet takes a few seconds to appear in all followers' feeds.

Estimation

DAU: 300 Million.
Tweets per day: 300M * 2 tweets/avg = 600 Million/day.
QPS (Write): 600M / 86400s \approx 7,000 tweets/sec.
QPS (Read): 300M DAU * 10 views/day = 3 Billion / 86400s \approx 35,000 queries/sec.
Storage: 600M tweets * 280 bytes \approx 168 GB/day.
Fan-out Load: If avg followers = 200, then 7k writes/sec becomes 1.4M "deliveries"/sec to timeline caches.

Blueprint

Concise Summary: A microservices-based architecture utilizing a "Write-path Fan-out" strategy. Tweets are ingested via a Tweet Service, queued for processing, and then pushed into the pre-computed Redis timelines of active followers.
Major Components:
Tweet Service: Handles incoming tweet creation and persistence.
Social Graph Service: Manages follow/unfollow relationships and queries follower lists.
Fan-out Worker: Asynchronously distributes tweet IDs to the cached timelines of followers.
Timeline Service: Serves the pre-computed Home and User timelines to the client.
Metadata DB (PostgreSQL): The source of truth for users, follows, and tweets.
Timeline Cache (Redis): High-speed storage for pre-computed user feeds.
Simplicity Audit: This design uses a standard asynchronous fan-out pattern. It avoids complex machine-learning ranking for the MVP, using purely chronological sorting to ensure speed and simplicity.

High Level Architecture

Sub-system Deep Dive

Service

Topology & Scaling: All services are stateless Golang/Java containers deployed in Kubernetes, allowing horizontal scaling based on CPU/Request count.
API Spec:
POST /v1/tweet: Body {text: string}. Returns tweet_id.
POST /v1/follow/{user_id}: Updates social graph.
GET /v1/timeline/home: Returns list of Tweet objects. Supports pagination via max_id.

Storage

Data Model:
Tweets Table: tweet_id (PK), user_id, content, created_at. Indexed on user_id for User Timelines.
Follows Table: follower_id, followee_id, created_at. Composite PK (follower_id, followee_id) for fast lookups.
Database Logic: Sharding by user_id across multiple PostgreSQL instances to prevent a single DB bottleneck.

Cache

Data Structures: Redis Lists or Sorted Sets per user_id. Key: TL:{user_id}, Value: [tweet_id_1, tweet_id_2, ...].
TTLs: 30-day TTL for active users. Inactive users' timelines are evicted and rebuilt from the Database upon their next login.
Eviction: LRU (Least Recently Used) policy.

Messaging

Topic Structure: tweet-published topic.
Delivery Guarantees: At-least-once delivery. Idempotency is handled at the worker level using the tweet_id.
Consumers: Fan-out workers consume from this topic to begin the distribution process.

Data Processing

Fan-out Worker Logic:
Fetch list of follower_ids for the tweet author from Social Graph DB.
For each follower_id, insert the tweet_id into their Redis Timeline List.
If follower_count > 100,000, mark the author as a "Celebrity" and skip the push (Celebrity tweets are fetched at read-time).
Wrap Up

Advanced Topics

Trade-offs: We prioritize Availability and Latency over Consistency (AP in CAP). A user might not see a friend's tweet for 1-2 seconds, which is acceptable.
Bottlenecks: The Fan-out for "Mega-influencers." If Elon Musk tweets, pushing to 100M+ Redis keys is too slow.
Failure Handling:
DB Replica: PostgreSQL uses Lead-Follower replication for read-scaling.
Redis Clusters: Redis is sharded and replicated to ensure feed availability.
Alternatives & Optimization:
Alternative: Using a Graph Database (Neo4j) for the social graph. Decision: Stick to PostgreSQL for MVP as many-to-many joins are performant enough with proper indexing at this scale.
Optimization: Use Pull model for celebrities. When a user requests their feed, the Timeline Service merges the Redis pre-computed feed with the latest tweets from any "Celebrities" the user follows.