The Question
Design

Scalable Social Media Photo Sharing Service

Design a mobile-first social photo-sharing application. The system must support photo uploads, a social graph (following/followers), and a personalized home feed showing recent content from followed users. Key challenges include handling high read-to-write ratios, managing the 'fan-out' of content to millions of followers, and ensuring low latency for global content delivery. Discuss trade-offs between push and pull models for feed generation and how to handle celebrity users with massive follower counts.
PostgreSQL
Redis
S3
CloudFront
RabbitMQ
Kubernetes
JWT
CDN
Questions & Insights

Clarifying Questions

What is the expected scale (DAU and total users)?
Assumption: 10 million registered users, 1 million Daily Active Users (DAU).
What is the typical user behavior (Read/Write ratio)?
Assumption: Heavy read-to-write ratio (100:1). Users view feeds much more than they post.
Should the feed be chronological or algorithmic?
Assumption: For the MVP, a simple reverse-chronological feed is sufficient.
What are the constraints on media?
Assumption: Photos only (JPEG/PNG), max 5MB per upload.
What is the consistency requirement for the "Follow" action vs the "Feed"?
Assumption: "Follow" should be highly consistent; "Feed" can be eventually consistent (a few seconds delay is acceptable).

Thinking Process

Core Bottleneck: The primary challenge is the "Fan-out" problem—efficiently delivering a single photo upload to thousands or millions of followers' feeds without high latency.
Key Questions:
How do we store and serve high-resolution photos globally with low latency?
How do we model the social graph to allow fast "Following" and "Follower" lookups?
Should we use a "Push" (Fan-out on write) or "Pull" (Fan-out on load) model for the feed?
How do we ensure the system remains available if the feed generation worker lags?

Bonus Points

Hybrid Fan-out Strategy: Implementing a "Push" model for regular users and a "Pull" model for "Celebrities" (high-follower accounts) to prevent "hotkey" issues in Redis.
CDN Edge Optimization: Using Signed URLs for secure photo uploads and CloudFront/Cloudflare for edge-caching content to reduce egress costs and latency.
Write-Through Caching: Using Redis not just for the feed but as a look-aside cache for user profiles and social graph metadata to hit sub-10ms response times.
Storage Sharding: Partitioning the Social Graph DB by user_id to handle scale beyond a single RDS instance.
Design Breakdown

Functional Requirements

Core Use Cases:
Users can upload photos.
Users can follow/unfollow other users.
Users can view a reverse-chronological feed of photos from people they follow.
Scope Control:
In-scope: Photo storage, Social graph management, Feed generation.
Out-of-scope: Comments, Likes, Real-time notifications, Algorithmic discovery, Direct Messaging.

Non-Functional Requirements

Scale: Support 1M DAU and 100M+ total photos.
Latency: Feed retrieval under 200ms; Photo upload under 2 seconds (excluding network transit).
Availability & Reliability: 99.9% uptime (CAP: High Availability over Strong Consistency).
Consistency: Eventual consistency for feed updates; Strong consistency for social graph (prevents "ghost" follows).
Security: Private photo access via pre-signed URLs; TLS for all transit.

Estimation

Traffic Estimation:
1M DAU * 1 upload/day = 11.5 Photos/sec (Avg). Peak = ~50/sec.
1M DAU * 20 feed reads/day = 230 QPS (Avg). Peak = ~1k QPS.
Storage Estimation:
1M photos/day * 2MB (avg) = 2TB/day. 730TB/year.
Bandwidth Estimation:
Inbound: 50 photos/sec * 2MB = 100MB/s.
Outbound: 1k QPS 10 photos 0.5MB (optimized) = 5GB/s (heavily handled by CDN).

Blueprint

Concise Summary: A microservices-based architecture utilizing an asynchronous "Push" model for feed delivery. Photos are stored in Object Storage, while the social graph and metadata reside in a Relational Database, with a Redis-based cache for rapid feed retrieval.
Major Components:
API Gateway: Entry point for authentication, rate limiting, and request routing.
Media Service: Handles photo metadata and coordinates uploads to Object Storage.
Social Service: Manages follow relationships and social graph queries.
Feed Service: Aggregates content and serves the user's personalized timeline.
Fan-out Worker: Asynchronously pushes new post IDs to followers' timelines.
Simplicity Audit: This design avoids complex stream processing or graph databases in favor of standard RDS and Redis, which are sufficient for 1M DAU and minimize operational overhead for an MVP.
Architecture Decision Rationale:
Push Model: Pre-computing feeds into Redis ensures that "Read Feed" (the most frequent operation) is a simple O(1) or O(N) fetch from a list.
Object Storage: S3 is the industry standard for durability and cost-effective photo storage.
Relational DB: PostgreSQL/MySQL handles the relational nature of follows and metadata with strong ACID guarantees.

High Level Architecture

Sub-system Deep Dive

Edge (Optional)

Content Delivery & Traffic Routing:
CloudFront CDN: Used to serve photos. Photos are resized at the edge or stored in multiple resolutions to save bandwidth.
Global Accelerator: Used for API routing to the nearest regional load balancer.
Security & Perimeter:
API Gateway: Performs JWT validation and Rate Limiting (e.g., 5 uploads/minute/user) to prevent abuse.
WAF: Protects against common web exploits.

Service

Topology & Scaling:
Stateless services deployed on Kubernetes (EKS) with Horizontal Pod Autoscaling (HPA) based on CPU/Request count.
API Schema Design:
POST /v1/photos: Upload photo. Returns photo_id.
POST /v1/social/follow/{user_id}: Follow a user. Idempotent.
GET /v1/feed: Returns a list of photo metadata and URLs. Supports pagination via cursor.
Resilience & Reliability:
Retries with exponential backoff for the Fan-out workers.
Circuit breakers on the Social Service to prevent Feed Service failures if the graph DB is slow.

Storage

Access Pattern:
Social Graph: High read/write (follows).
Metadata: High write (on upload), high read (on feed load).
Database Table Design:
Users: id, username, email, created_at
Photos: id, user_id, s3_url, created_at
Follows: follower_id, followee_id, created_at (Unique Index on pair).
Technical Selection:
PostgreSQL: Robustness and support for complex JOINs needed for friend-of-friend queries if the MVP grows.
Distribution Logic:
Shard Follows table by follower_id to ensure all "Who do I follow?" queries hit a single shard.

Cache

Purpose & Justification: Reduces Feed read latency from ~500ms (SQL Join) to <20ms (Redis fetch).
Key-Value Schema:
Key: feed:{user_id}
Value: Redis LIST or ZSET of photo_ids.
TTL: 72 hours (Inactive users don't need cached feeds).
Failure Handling: If Redis is down, the Feed Service falls back to querying the SQL Metadata DB directly (Degraded performance).

Messaging

Purpose & Decoupling: Decouples the photo upload completion from the feed update process.
Event Schema: { "photo_id": "123", "author_id": "456", "timestamp": "..." }
Technical Selection: RabbitMQ or AWS SQS for simplicity and low operational cost for MVP scale.

Data Processing

Processing Model: The Fan-out Worker is a consumer that:
Fetches all follower_ids for the author_id from the Social Service.
Updates each follower's feed:{user_id} list in Redis.
Scalability: For "Celebrities", the worker skips the push and the Feed Service "Pulls" their photos at read-time (Hybrid approach).
Wrap Up

Advanced Topics

Trade-offs: We choose Eventual Consistency for the feed. A user might not see their friend's post for 1-2 seconds, but this allows the system to remain highly responsive and handle write spikes.
Bottleneck Analysis: The "Celebrity" problem is the main risk. If a user with 10M followers posts, the Fan-out worker could overwhelm Redis. Mitigation: Use a threshold (e.g., >50k followers = Pull Model).
Security & Privacy: All photo URLs in S3 are private. The Media Service generates Presigned URLs with a short expiry (e.g., 1 hour) when the feed is requested, ensuring photos cannot be scraped easily.
Distinguishing Insights: For a truly "simple" MVP, one could argue for a "Pull" model (SQL JOIN at read time) to avoid Redis entirely. However, even at 1M DAU, the JOIN performance on a large Follows table degrades quickly, making the Redis "Push" model a more staff-level scalable recommendation.