The Question
ML DesignReal-Time Video Anomaly Detection System
Design a high-scale, real-time video surveillance system capable of detecting anomalous activities across 10,000+ camera streams. The system must process high-resolution temporal data with sub-2-second end-to-end latency. Detail your approach to handling massive data throughput, the choice of ML paradigm (supervised vs. unsupervised) given the rarity of anomaly labels, the trade-offs between edge and cloud compute, and how you ensure system reliability and model freshness in changing physical environments (e.g., lighting, weather).
3D-Convolutional Autoencoder
Vision Transformer
TensorRT
Kafka
Flink
OpenVINO
PyTorch
S3
GMM
ONNX
Questions & Insights
Clarifying Questions
Business Goal: Is the priority "High Recall" (don't miss any crime/accident) or "High Precision" (minimize false alarms for security staff)?
Assumption: High Recall is prioritized for safety, with a target of < 5% False Discovery Rate (FDR).
Constraints & Scale: How many cameras are we supporting, and what is the resolution/FPS?
Assumption: 10,000 concurrent camera streams, 1080p resolution, sampled at 5-10 FPS for anomaly detection.
Edge vs. Cloud: Is compute happening on the camera (edge), a local gateway, or the cloud?
Assumption: A hybrid approach. Edge devices perform basic motion filtering; heavy ML inference happens in a regional cloud/private data center for cost-efficiency and model complexity.
Definition of Anomaly: Are we looking for specific actions (violence, falling) or general deviations from "normal" patterns?
Assumption: General deviations (unsupervised/semi-supervised) to capture "unknown unknowns," with a small set of supervised classifiers for high-value events (e.g., fire, intrusion).
Latency Budget: What is the end-to-end P99 latency from event occurrence to alert?
Assumption: P99 < 2 seconds.
Thinking Process
Identify the Bottleneck: Raw video data is massive. Transferring 10k 1080p streams to the cloud is cost-prohibitive. I need a Cascaded Inference strategy: Simple motion detection on the edge -> Feature extraction -> Complex anomaly scoring in the cloud.
Choose the ML Task: Since anomalies are rare (class imbalance), supervised learning will fail on "novel" anomalies. I will frame this as a Semi-supervised Reconstruction Task or Self-supervised Temporal Prediction. If the model can't "reconstruct" or "predict" the next N frames, it's an anomaly.
Temporal Modeling: Spatial features (single frame) aren't enough. I need 3D-Convs or Transformers to capture motion dynamics (temporal features).
Scaling the Solution: Use a message bus (Kafka) to decouple frame ingestion from inference workers. Use specialized hardware (GCP TPUs or AWS Inferentia) for the reconstruction model.
Elite Bonus Points
Spatio-Temporal Consistency: Implementing a "smoothing" window to ensure an anomaly persists for N frames before alerting, reducing "glitch-induced" false positives.
Continual Learning with Feedback Loop: Implementing an active learning UI where security guards mark alerts as "True/False Anomaly," which triggers high-priority fine-tuning of the reconstruction threshold.
Privacy-Preserving Inference: Utilizing on-device PII masking (face blurring) before the video stream leaves the local network to comply with GDPR/CCPA.
Model Quantization (FP16/INT8): Using TensorRT or OpenVINO to optimize the backbone for low-latency inference on the serving layer.
Design Breakdown
Requirements
Product Goal: Detect and alert security personnel to unusual activities in real-time.
Success Metrics:
Online Metrics: Mean Time to Detect (MTTD), Alert Precision@K, Daily False Alarms per Camera.
Offline Metrics: ROC-AUC on a curated "anomaly" test set, Reconstruction Error distribution.
Guardrail Metrics: P99 Latency, Network Bandwidth usage, System uptime (99.9%).
System Constraints: 10k cameras, ~100k FPS total throughput, 2s latency SLA.
Data Availability: Petabytes of "normal" footage; very sparse (labeled) anomaly footage.
ML Problem Framing
ML Task Type: Semi-supervised Anomaly Detection (Reconstruction-based).
Prediction Target: Minimize reconstruction error L = ||x - \hat{x}||^2 for normal video patches.
Inputs:
User/Context: Camera ID, Location (e.g., "parking lot" vs "lobby"), Time of day.
Item (Video): Sequence of T frames (e.g., 16-32 frames).
Outputs: Anomaly Score S \in [0, 1].
ML Challenges: Extreme Data Imbalance, Environment Drift (lighting changes, weather), and High Dimensionality.
Design Summary & MVP
Concise Summary: A cascaded architecture where edge devices filter static scenes, and a Cloud-based Convolutional Autoencoder (CAE) processes video clips to detect high reconstruction errors.
Model Architecture & Selection:
Baseline Model: Simple background subtraction + thresholding on pixel-change count.
Target Model: 3D-Convolutional Autoencoder (3D-CAE) or a Vision Transformer (ViT) Masked Autoencoder.
Choice Rationale: 3D-CAE captures both spatial (objects) and temporal (motion) features. Using an autoencoder allows us to train on the abundant "normal" data.
Simplicity Audit: We avoid Reinforcement Learning or complex Graph Neural Networks initially. An Autoencoder on video patches is the most robust MVP for "unseen" anomalies.
System Architecture
Pipeline Deep Dive
Data Pipeline
Data Source: 10,000 RTSP/H.264 streams.
Data Ingestion: Edge devices (cameras/NVRs) run a lightweight motion mask (Gaussian Mixture Model). If motion > threshold, frames are sent to Kafka/Kinesis. This reduces cloud ingress by ~70%.
Data Storage: Raw "anomalous" clips stored in S3 with lifecycle policies. Normal data is downsampled and stored for retraining.
Data Processing: Flink or Spark Streaming handles the sliding window buffer (grouping frames into 2-second clips).
Feature Pipeline
Feature Engineering:
Normalization: Zero-centering and unit variance per channel.
Spatial Downsampling: Resize 1080p to 224x224 to fit model input.
Temporal Subsampling: Sample 8-16 frames per clip to reduce redundancy.
Online Pipeline: Low-latency decoding using NVDEC (NVIDIA Decoder) to keep the GPU dedicated to inference.
Training/Serving Skew: A shared library for frame pre-processing (OpenCV/Torchoision) is used in both training (offline) and inference (online) to ensure identical pixel values.
Model Architecture
Problem Formulation: Unsupervised reconstruction.
Target Model: 3D Convolutional Autoencoder (3D-CAE).
Encoder: Series of 3D-Conv layers + MaxPool3D to compress the clip into a latent bottleneck.
Decoder: 3D-Deconv (Transpose) layers to reconstruct the original clip.
Rationale: Normal behaviors (walking, standing) are easily reconstructed (low MSE). Anomalies (running, fighting, falling) are "out of distribution" for the model, leading to high MSE.
Optimization: Quantization to INT8 and Pruning of the encoder layers since we need high-throughput serving for 10k cameras.
Training Pipeline
Dataset Construction: Use 1 week of "clean" footage (verified no incidents).
Data Splitting: Time-based split. Train on Week 1, Validate on Week 2, Test on known anomaly clips.
Training Infrastructure: Distributed Data Parallel (DDP) on a GPU cluster.
Retraining: Scheduled monthly or if Population Stability Index (PSI) of the reconstruction error shifts significantly (e.g., due to seasonal changes like snow).
Serving Pipeline
Serving Pattern: Request-Response via gRPC or high-throughput Stream Processing.
Latency Optimization:
Batching: Aggregate clips from multiple cameras into a single GPU batch.
Model Partitioning: Use a small "Student" model for common cameras and a large "Teacher" model for high-security zones.
Reliability: If the ML service fails, fallback to simple motion-based alerting (fail-safe).
Evaluation Pipeline
Offline Evaluation: Area Under the ROC Curve (AUROC) on a benchmark dataset (like UCF-Crime or internal gold sets).
Online Evaluation: Precision@Top-K alerts. If security guards click "Ignore" on the first 3 alerts, the model precision is low.
Monitoring Pipeline
Data Monitoring: Track "Average Pixel Intensity" to detect camera tampering or hardware failure (black screen).
Model Monitoring: Monitor the distribution of Anomaly Scores. If the mean score jumps globally, there's likely a data pipeline or lighting issue, not 10,000 simultaneous crimes.
Wrap Up
Final Evaluation
Trade-offs:
Accuracy vs. Latency: Larger 3D-Conv kernels improve accuracy but increase FLOPs. We use Separable 3D Convolutions to strike a balance.
Complexity vs. Maintainability: An unsupervised AE is easier to maintain than 50 different supervised "action" classifiers.
Edge Cases:
Cold Start: For a new camera location, use a "Global" model for 24 hours, then fine-tune a "Local" weights branch to learn that specific background.
Adversarial: Use robust feature extraction to prevent "cloaking" (e.g., wearing patterns that confuse the autoencoder).