Scalable Unified Data Platform (Batch & Streaming)

Scalable Unified Data Platform (Batch & Streaming)

Design a high-throughput data platform capable of processing 100TB of daily data via both batch and streaming jobs. The system must support 'Exactly-Once' processing semantics, unified metadata management for cross-job consistency, and a 'Lakehouse' storage model to handle ACID transactions on object storage. Address how the platform will manage the 'small files' problem inherent in streaming ingestion and ensure cost-efficient resource scaling during peak loads.
Apache SparkApache KafkaApache IcebergS3KubernetesApache AirflowTrinoAvroPrometheusOpenLineage
00
Read
1
InterviewGPT

AI-powered tools to help you succeed in tech interviews — from resume to offer.

Interview Solver

  • Coding Puzzles
  • System Design
  • Behavioral Challenges
  • ML System Design
  • SQL Puzzles
  • FE System Design
Explore Solver

Question Bank

  • Coding Interview Questions
  • System Design Interview Questions
  • Behavioral Interview Questions
  • ML System Design Questions
  • SQL & Database Questions
  • FE System Design Questions
Explore Questions

Golden Blogs

  • Coding Solutions
  • System Design Guides
  • Behavioral Guides
  • ML System Design Guides
  • SQL Solutions
  • FE System Design Guides
Explore Blogs

Intervipedia

  • Coding Concepts
  • System Design Concepts
  • Behavioral Concepts
  • ML System Concepts
  • SQL Concepts
  • FE System Concepts
Explore Concepts

Application Tools

  • Self-Intro Generator

Company

  • Pricing
  • FAQ
  • About
  • Privacy Policy
  • Terms of Service

© 2026 InterviewGPT Inc. All rights reserved.

All systems operationalUS-East

Made with ♥ for developers