The Question
ML Design

Scalable Multi-Modal Content Moderation System

Design a high-scale content moderation system for a global social media platform capable of processing billions of multi-modal posts per day. The system must automatically identify and action policy-violating content (e.g., hate speech, NSFW) in real-time while minimizing false positives and incorporating a human-in-the-loop workflow for high-uncertainty cases.
Multi-Modal Transformer
BERT/LLM
CNN
MTL
Active Learning
Questions & Insights

Clarifying Questions

Business Goal: Is the primary goal to reduce "Harmful Content Prevalence" (the % of views on violating content) or to minimize "Time to Takedown"? (Assumption: Both, but Prevalence is the North Star).
Constraints & Scale: What is the volume? (Assumption: 1 Billion posts/day, 50k QPS peak). What is the latency budget? (Assumption: <100ms for synchronous "blocking" moderation, <2s for asynchronous "post-publish" moderation).
Edge Cases: How do we handle multi-modal content (text + image + video)? How do we deal with "adversarial" users who slightly modify prohibited content to bypass filters?
Assumptions:
Multi-modal input (Text + Images).
Policy includes NSFW, Hate Speech, Violence, and Self-harm.
Tiered moderation: Automated blocking, Shadow-banning, and Human-in-the-loop (HITL) review.
Global scale requiring multi-lingual support.

Thinking Process

The Funnel Approach: A single massive model is too slow for 50k QPS. I need a "Cascade" architecture: fast heuristics/hash-matching first, then lightweight text/image classifiers, then a heavy multi-modal fusion model for the "gray area" content.
Addressing Data Drift: Moderation is a "Red Queen's Race." Bad actors change tactics daily. I must prioritize a system that allows for rapid model retraining and a tight feedback loop from human moderators.
High Recall vs. High Precision: For harmful content like self-harm, recall is vital. For controversial but non-violating content, precision is vital to avoid over-censorship. I should propose a multi-threshold strategy.
Embedding Space: Use a shared multi-modal embedding space (like CLIP) to catch "cross-modal" violations (e.g., a benign image with a hateful caption).

Elite Bonus Points

Adversarial Robustness: Implementing "Proactive Detection" by training on adversarially perturbed data (e.g., text with typos like "H4te", or images with minor pixel noise).
Active Learning with "Uncertainty Sampling": Instead of random human review, send samples where the model's Softmax output is near 0.5 to humans to maximize the "information gain" for the next training iteration.
Graph-based Context: Utilizing a "User Reputation Graph." If a known bad actor posts a borderline image, the threshold for deletion should be lower than for a high-reputation user.
Calibration for Policy Changes: When legal policies change (e.g., new local laws), use "Isotonic Regression" or Platt Scaling to recalibrate model scores to new decision boundaries without retraining the entire backbone.
Design Breakdown

Functional Reqs

Real-time Inference: Content is scanned upon upload.
Action Engine: Automatically block, flag for review, or apply "soft-interventions" (warning labels).
Appeals Process: Users can appeal a takedown, triggering a high-priority human review.
Audit Log: Every action (AI or Human) must be logged for transparency and legal compliance.

Non-Functional Reqs

Latency: <200ms for automated decisions to ensure a smooth UX.
Scalability: Must handle viral events (e.g., sports, elections) where traffic spikes 10x.
Freshness: Ability to push "emergency" keyword/hash lists to production in <5 minutes.
Availability: 99.99% (Moderation is a "Tier 0" service; if it fails, the app must either "fail-safe" or "fail-open" depending on risk).

ML Problem Framing

ML Objective: Multi-label classification. For a post Predict a vector \mathbf{y} \in [0, 1]^K where K is the number of policy categories.
ML Category: Supervised learning (classification) + Representation learning (Embeddings).
Input/Output/Label:
Input: Raw text, Image pixels, User metadata, Contextual signals.
Output: Probability scores per category + Embedding vector.
Labels: Binary flags from human moderators (1 = Violation, 0 = Safe).

Data Prep & Features

Data Pipeline:
Signals: Perceptual hashes (pHash/dHash) for images to catch exact/near duplicates.
Text: Tokenization (BPE/WordPiece), removal of "leetspeak" via normalization.
Feature Engineering:
User Features: Historical violation rate, account age, verification status.
Content Features: Multi-modal embeddings (CLIP-like), OCR (text inside images), object detection (e.g., detecting weapons).
Context Features: Community/Subreddit norms, geolocation (local laws).
Feature Store: Store pre-computed user embeddings and "Known Bad" hash-sets for O(1) lookup.

Model Architecture

Level 1 (Fast Path): Deterministic checks (RegEx, Hash-matching) + Lightweight DistilBERT/FastText.
Level 2 (Deep Ranking):
Text Tower: RoBERTa or XLM-R (for multi-lingual).
Vision Tower: ViT (Vision Transformer) or EfficientNet.
Fusion Layer: A Cross-Attention mechanism that looks for interactions between text and image (e.g., "Visual-Linguistic BERT").
Loss Function: Focal Loss to handle extreme class imbalance (99% of content is safe).

Training & Serving

Optimization: Distributed training using Horovod/DeepSpeed on GPU clusters.
Serving: Hybrid approach. Level 1 runs on CPU-based edge nodes. Level 2 runs on GPU clusters (Triton Inference Server) with dynamic batching.
Training/Serving Skew: Use a "Log-and-Wait" strategy where features used for inference are logged to the training set to ensure the model trains on exactly what it sees at runtime.

System Architecture

Pipeline Deep Dive

Data Pipeline

Ingestion: Kafka handles the firehose of 1B events/day. Events are partitioned by user_id to maintain causal ordering.
Data Storage: S3 acts as the "Data Lake." We store raw content and a "Silver Layer" of normalized metadata for 90 days for audit/retraining.

Feature Pipeline

Streaming Features: Flink calculates real-time aggregates like "User violation rate in the last 10 minutes" to catch spam bots.
Consistency: We use a unified Feature Store (e.g., Tecton or Feast). During training, we perform "Point-in-time" joins to avoid data leakage from the future.

Training Pipeline

Workflow: Airflow orchestrates daily retraining.
Stratified Sampling: Since violations are rare, we upsample "Safe" content that was "Close to the boundary" (Hard Negatives) to improve the model's discriminative power.

Serving Pipeline

Retrieval/Filtering: Before the ML model, we use "Bloom Filters" for billion-scale prohibited hashes to save compute.
Re-ranking/Policy: The final layer isn't just a model score; it's a logic gate. For example, "If (Hate_Speech > 0.9) OR (Violence > 0.8 AND User_Reputation < 0.2), then Delete."

Evaluation Pipeline

Interleaved Testing: For moderation, we sometimes use "Shadow Mode" where the new model predicts but doesn't act, and we compare its agreement with the existing production model.
Feedback: Human labels are the "Gold Standard." We use "Consensus Scoring" (3 moderators per item) to ensure label quality.

Monitoring Pipeline

Feature Drift: Monitor the distribution of input embeddings using JS Divergence. If the "Centroid" of the text embeddings shifts significantly, it indicates a new viral topic or slang.
Model Staleness: If the F1-score of the model against human labels drops below a threshold, trigger an emergency retraining.
Wrap Up

Advanced Topics

Offline Metrics: Area Under Precision-Recall Curve (AUPRC) because of class imbalance. We also track "Calibration Error."
Online Metrics:
Prevalence: % of content views that violate policy.
Auto-action Rate: % of content handled without human intervention.
Appeal Overturn Rate: A measure of False Positives.
Deployment:
Shadow Mode: Run Level 2 model in the background.
Canary: Enable for 1% of traffic.
Full Rollout: Monitor P99 latency.
Failure Modes: If Triton times out, fall back to the Level 1 Fast Filter (Safe-fail). If a malicious "injection" is detected in text, trigger immediate human review.
Responsible AI: Perform "Counterfactual Fairness" checks—ensure the model doesn't disproportionately flag specific dialects (e.g., AAVE) or religious terms when used in a benign context.