Design a high-throughput, low-latency recommendation engine for a global video sharing platform. The system should handle hundreds of millions of users and items, provide real-time personalization based on user behavior, and optimize for long-term engagement metrics like retention and watch time, while ensuring diversity and handling new content discovery.