CCBot — CCBot is the crawler of Common Crawl, a nonprofit that publishes a large open archive of the web. Many AI/SI models have been trained on Common Crawl data. Blocking CCBot keeps future pages out of that archive. It is a frequent entry in robots.txt files that limit AI training.
Added October 21, 2026

