The Question
ML DesignInvestor Quality Ranking System
Design a high-scale ranking and discovery system for a fintech platform to identify and surface 'high-quality' investors. The system must process millions of portfolios and trade logs to distinguish between skill-based performance and random luck (survivorship bias). Focus on the end-to-end ML lifecycle: from real-time feature engineering of risk-adjusted metrics to a two-stage retrieval and ranking architecture that meets low-latency requirements for a social discovery feed. Address challenges such as financial non-stationarity, market regime shifts, and the cold-start problem for new accounts, while ensuring the serving infrastructure is robust enough for high-concurrency traffic.
LightGBM
Kafka
Flink
Spark
Tecton
Sharpe Ratio
Bayesian Shrinkage
Feature Store
Ray
MLflow
Questions & Insights
Clarifying Questions
Business Goal: Is the goal to create a social "copy-trading" feature, a leaderboard, or internal risk management?
Assumption: The goal is to identify and rank high-quality investors to feature them in a "Discover" feed to improve platform engagement and financial literacy.
Defining "Good": How do we define a good investor? Is it raw ROI, risk-adjusted returns, or consistency?
Assumption*: We define a "good investor" using the Sharpe Ratio** (excess return per unit of volatility) over a rolling 12-month window to filter out "lucky" gamblers.
Constraints & Scale: What is the scale of the user base and the latency requirements?
Assumption: 25M monthly active users (MAU), 100M+ total accounts. The system must support a QPS of 5k for the discovery feed with a P99 latency of <150ms.
Data Freshness: How quickly should an investor's rank update after a trade?
Assumption: Portfolio values update in near real-time (streaming), but the global "good investor" model updates daily (batch).
Thinking Process
Identify the Core Challenge: The main hurdle is distinguishing "skill" from "luck." Financial data is extremely noisy. A user might have 1000% returns by betting on a single meme stock (low skill, high luck), which is a bad recommendation for others.
Two-Stage Approach: Given 25M+ users, I can't rank everyone in real-time. I need a Retrieval step (filter for active, diversified, positive-return users) followed by a Ranking step (pointwise or pairwise model to predict long-term performance consistency).
Metric Selection: Raw ROI is a dangerous label. I will use a multi-objective approach or a risk-adjusted metric like Sharpe or Sortino ratio as the primary regression target.
Scale and YAGNI: I'll start with a simple XGBoost model on historical features rather than a complex Transformer. The bottleneck will be feature engineering (aggregating trade history), not the model architecture.
Elite Bonus Points
Bayesian Shrinkage for Small Sample Sizes: For new investors with few trades, their ROI is highly volatile. I would apply empirical Bayes to "shrink" their scores toward the platform mean until they have more history, preventing "lucky beginners" from topping the charts.
Survivorship Bias Mitigation: I will include closed accounts or "failed" investors in the training set to ensure the model learns the features of poor investing (e.g., lack of diversification, high churn) and doesn't just overfit to current winners.
Handling Non-Stationarity: Market regimes change (Bull vs. Bear). I'll implement "Market Context" features (VIX index, S&P 500 trend) so the model understands that a 5% return in a crashing market is better than a 10% return in a mooning market.
Look-ahead Bias Prevention: Rigorous time-based splitting is required. I must ensure that features at time T only use information available before To predict performance at T+1.
Design Breakdown
Requirements
Product Goal: Identify and rank investors who exhibit sustainable, high-quality investment strategies for discovery.
Success Metrics:
Online: Increase in "Follow" rate on discovery profiles, 3-month retention of followers, and average Sharpe ratio of recommended investors.
Offline: Precision@K, NDCG for ranking, and Mean Absolute Error (MAE) for ROI prediction.
Guardrail: Portfolio turnover rate of recommended investors (to avoid promoting day-trading).
System Constraints: Daily batch retraining, real-time feature serving for market price updates, 150ms P99 latency.
Data Availability: Trade execution logs, daily portfolio snapshots, user profile metadata, and real-time market data (NBBO).
ML Problem Framing
ML Task Type: Ranking (Learning to Rank). Specifically, a pointwise regression approach to predict a "Quality Score" (Sharpe Ratio).
Prediction Target: y = \text{Sharpe Ratio}_{t \to t+30d}. We predict the risk-adjusted performance for the next 30 days.
Inputs:
User Features: Account age, verification status, self-reported experience.
Behavioral (Item) Features: Diversification score (HHI index), trading frequency, max drawdown, average holding period, sector exposure.
Context: Market volatility, current interest rates, sector-specific momentum.
Outputs: A scalar quality score used to rank candidates in the discovery feed.
ML Challenges: High noise-to-signal ratio, extreme outliers (meme stock winners), and the need for "Explainability" (Why is this investor "good"?).
Design Summary & MVP
Concise Summary: A two-stage system: 1) Heuristic-based retrieval to filter for active, diversified users, and 2) A LightGBM regressor that predicts the 30-day forward-looking Sharpe Ratio based on historical portfolio statistics.
Baseline Model: A heuristic ranker: \text{Rank} = \text{ROI} \times \log(\text{Days Active}).
Target Model: LightGBM (Gradient Boosted Decision Trees).
Choice Rationale: Tabular financial data is non-linear and has many missing values. GBDTs outperform Deep Learning on medium-sized tabular datasets and are significantly cheaper to train and serve.
Simplicity Audit: We avoid Deep Learning/Transformers initially because the primary gains come from robust feature engineering (risk metrics) rather than high-dimensional embeddings.
System Architecture
Pipeline Deep Dive
Data Pipeline
Data Source: Trade logs (Kafka), Portfolio snapshots (Postgres/S3), and Market prices (External API).
Data Ingestion: Hybrid approach. Trade events are streamed via Kafka for "fresh" ROI calculations. Portfolio snapshots are processed in daily batches via Airflow.
Data Storage: S3 Data Lake for raw logs (Parquet). Snowflake for structured features used in analysis.
Data Processing: Spark handles the heavy lifting of calculating the HHI index (diversification) and rolling volatility across 25M users.
Data Quality: We implement "Sanity Checks" (e.g., user ROI cannot be +1,000,000% unless it's a known data error or extreme outlier to be capped).
Feature Pipeline
User Features: Tenure, total deposits, self-identified "Aggressive" vs "Conservative" tag.
Portfolio Features:
Risk Metrics: Beta (correlation to S&P 500), Max Drawdown, Value at Risk (VaR).
Concentration: Number of tickers, % in top 3 holdings.
Consistency: Monthly ROI variance.
Online Feature Pipeline: Flink calculates "Intraday ROI" by joining trade events with real-time stock prices.
Feature Store: Use Tecton or Feast to ensure that the "Sharpe Ratio" calculated in training (Offline) is identical to the one used in serving (Online).
Model Architecture
Problem Formulation: Pointwise Regression. Predict y \in \mathbb{R} (Future Sharpe Ratio).
Candidate Models:
Linear Regression: Too simple, misses non-linear risk interactions.
Deep Learning (MLP): Overkills the problem and is harder to tune for tabular data.
LightGBM*: Selected.** Handles outliers via binning, natively supports missing values, and is highly efficient.
Architecture Design: Features are fed into a 500-tree LightGBM model.
Optimization: Use "Monotonic Constraints" for features like "Account Age" (older accounts are generally more stable).
Training Pipeline
Dataset Construction: We use a Sliding Window approach. Train on months 1-6 to predict month 7. Slide and repeat.
Labeling: The label is the Forward 30-day Sharpe Ratio.
Data Splitting: Time-series split is mandatory. If we train on 2023 data and test on 2022, we leak the "Bull Market" information.
Retraining: Weekly retraining via Airflow to capture new market trends.
Serving Pipeline
Serving Pattern: Two-stage.
Retrieval: SQL query on Feature Store:
SELECT user_id FROM features WHERE trades_count > 10 AND diversification_score > 0.2.Ranking: The top 1,000 candidates from retrieval are sent to the LightGBM model hosted on Seldon Core or AWS SageMaker.
Latency: We use a Model Cache (Redis) to store scores for the top 50,000 investors, as their scores don't change second-by-second.
Evaluation Pipeline
Offline: We look at the correlation between predicted Sharpe and actual Sharpe. We also measure Feature Importance to ensure the model isn't just picking people who own one lucky stock.
Online: Randomized A/B test. Treatment group sees the ML-ranked list; Control sees the heuristic (Raw ROI) list.
Monitoring Pipeline
Drift: Monitor the "Diversification Score" distribution. If the market crashes, users might sell everything, causing a feature shift.
Delayed Feedback: Financial labels take 30 days to realize. We use "Proxy Labels" (1-day performance) for early detection of model failure.
Wrap Up
Final Evaluation
Observability: We track the "Top 10" investors daily. If the same user stays there for months, the model is stable. If it rotates every hour, it's too noisy.
Edge Cases: Cold Start. New users get a "Neutral" score. The "GameStop" Effect: Sudden spikes in a single stock can create "fake" good investors. We mitigate this with a "Sector Concentration" penalty in the feature layer.
Trade-offs: We sacrifice raw performance (highest ROI) for stability (risk-adjusted). This might result in "boring" leaders, but it protects the platform from promoting reckless gambling.