robots.txt — robots.txt is a plain-text file at the root of a website that tells crawlers which parts of the site they may visit. It is how most website owners allow or block search engines and AI/SI crawlers. It works on an honor system, which major companies respect but rogue scrapers often ignore. It should always allow access to llms.txt.

Added October 21, 2026