Design a distributed system capable of crawling and indexing billions of web pages. The system must efficiently manage URL discovery, ensure politeness to host servers, handle content deduplication, and store massive amounts of unstructured data while maintaining high horizontal scalability.