Training crawler — A training crawler is an automated program that collects web content for use as AI/SI training data. Examples include GPTBot and CCBot. Blocking training crawlers keeps content out of future model training but does not remove what was already collected. A surgical block stops training crawlers while still allowing retrieval agents.
Added October 21, 2026

