Common Crawl — Common Crawl is a nonprofit that crawls the web and publishes a free archive of billions of web pages. Many AI models have been trained partly on its data. Its crawler is called CCBot. Website owners who want to stay out of future training data often block it.

Added October 21, 2026