Real-Time Financial Signal Generation System
Design a high-scale machine learning system to predict short-term equity price movements (e.g., 1-5 minute horizon) for a universe of 5,000 stocks. The system must process real-time market data feeds, perform feature engineering with minimal latency, and provide directional signals with a P99 latency of less than 50ms. Address specific challenges such as non-stationary data, prevention of look-ahead bias in the data pipeline, backtesting strategies, and the design of a robust online/offline feature store to maintain consistency.
LightGBMApache FlinkKafkaRedisOptunaRayParquet
00