The Question
Design

Photo & Video Sharing Social Network

Design a photo and video sharing social network similar to Instagram. The system should support media uploads with processing and CDN delivery, algorithmic feeds, stories with 24-hour expiry, direct messaging, and engagement features for hundreds of millions of daily active users.
PostgreSQL
S3
Redis
CDN
Kafka
Design Breakdown

Functional Requirements

Post Media: Users can upload photos (MVP focus) with captions.
Follow/Unfollow: Users can subscribe to other users' content.
Home Feed: Users see a chronological/ranked stream of photos from people they follow.
Media Viewing: High-performance retrieval of images.

Non-Functional Requirements

High Availability: Read availability is paramount; users should always see a feed.
Low Latency: Images must load quickly (CDN usage).
Data Durability: Uploaded media must not be lost.
Eventual Consistency: New posts may take a few seconds to propagate to all followers.

Estimation

DAU: 500 Million.
Write Path: ~100M posts/day. 100M / 86,400s \approx1,150 uploads/sec.
Read Path: ~50B feed views/day. 50B / 86,400s \approx580,000 QPS.
Storage: 100M images 2MB (avg size across variants) $\approx$ 200 TB/day**.
Bandwidth: 580k QPS 5 images * 200KB (thumbnails) $\approx$ 580 GB/s outgoing**.

Blueprint

Concise Summary: A media-centric architecture using a decoupled upload-processing pipeline and a pre-computed feed strategy.
Major Components:
Media Service: Manages multipart uploads and generates signed URLs for S3.
Image Processor: Asynchronous worker that creates thumbnails and optimizes images.
Metadata Service: Manages post details, hashtags, and media locations in a relational database.
Feed Service: Aggregates post metadata for user timelines.
Object Store (S3): The source of truth for all binary media files.
CDN: Distributes media to edge locations to minimize latency.
Simplicity Audit: This design uses S3 for durability and a simple async worker for image processing, avoiding the complexity of a real-time media transcoding cluster for the MVP.

High Level Architecture

Sub-system Deep Dive

Service

Topology & Scaling: Stateless microservices. The Media Service is scaled based on network I/O, while the Metadata/Feed services are scaled based on CPU.
API Spec:
POST /v1/media/upload: Returns a pre-signed S3 URL for direct client upload (minimizing server bandwidth).
POST /v1/posts: Links the uploaded S3 key with a caption and user metadata.
GET /v1/feed: Returns a list of post objects including CDN URLs.

Storage

Data Model:
Posts Table: post_id, user_id, s3_key_original, s3_key_thumb, caption, created_at.
Database Logic: PostgreSQL with partitioning on user_id. Indexes on created_at for feed ordering and user_id for profile views.
Blob Storage: Amazon S3 organized by /year/month/day/user_id/ for easy lifecycle management.

Cache

Data Structures: Redis Sorted Sets (ZSET). Key: feed:{user_id}, Member: post_id, Score: timestamp.
Logic: Stores the top 500 post IDs for each user's feed. When the feed is requested, the service fetches post IDs from Redis and hydrations them with metadata from PostgreSQL (or a secondary Metadata Cache).

Messaging

Structure: Kafka or RabbitMQ.
Topic: media_uploaded.
Consumer: The Image Processor listens for new uploads to trigger thumbnail generation (e.g., 200x200, 600x600 versions).
Wrap Up

Advanced Topics

Trade-offs:
Client-side vs Server-side Upload: Using pre-signed URLs (Client-side) reduces server load but makes it harder to validate file integrity before storage.
Availability vs. Consistency: In the event of a Redis failure, we fallback to a slow DB query, prioritizing availability over low latency.
Bottlenecks:
S3 Latency: S3 is not meant for direct high-frequency reads. The CDN is a mandatory optimization to prevent S3 from becoming a bottleneck and to reduce egress costs.
Fan-out: High-volume "Celebrity" posts can overwhelm the feed cache update workers.
Failure Handling:
Dead Letter Queues (DLQ): If the Image Processor fails to generate a thumbnail, the message is moved to a DLQ for manual inspection or retry.
Multi-Region S3: Replication across regions for disaster recovery.
Alternatives:
NoSQL (Cassandra): Could be used for the metadata layer to handle massive write scales, but PostgreSQL is chosen for MVP due to easier indexing and schema flexibility.