Design a web crawler.

Focus on the architecture and components of the web crawler. Start by outlining the core functionalities, such as URL fetching, parsing, and storing data. Discuss how you would handle challenges like duplicate URLs, politeness policies (respecting robots.txt), and rate limiting to avoid overwhelming servers. Consider scalability and how you would manage a large volume of data, including storage solutions and database choices. Finally, think about how to ensure the crawler can adapt to changes in website structures and content.