Scalable Centralized Logging and Observability Platform

Design a high-scale centralized logging system capable of ingesting 100 TB of data per day from thousands of microservices. The system must support real-time full-text search for the most recent 7 days of logs and provide cost-effective archival for up to 30 days. Address how the system handles extreme traffic spikes, ensures minimal impact on application performance, and manages the operational complexity of large-scale indexing. Discuss trade-offs between ingestion latency, search performance, and storage costs.

KafkaOpenSearchS3FluentBitgRPCProtobufGo

Read

Global Large-Scale Log Aggregation System

Design a highly available and cost-effective log collection system for a global infrastructure spanning 10 data centers with over 1 million servers. The system must handle a combined throughput of 10 PB of log data per day, ensuring logs are searchable within one minute while providing long-term archival. Address challenges regarding cross-region network latency, data durability during partitions, and the cost trade-offs between real-time indexing and cold storage.

KafkaOpenSearchS3VectorFluent BitGogRPCZstandardAthenaTLS 1.3

Read

Scalable Centralized Logging and Observability Platform

KafkaOpenSearchS3FluentBitgRPCProtobufGo

Read

Global Large-Scale Log Aggregation System

KafkaOpenSearchS3VectorFluent BitGogRPCZstandardAthenaTLS 1.3

Read