User agent — A user agent is the name a browser or bot announces when it requests a web page. Website owners use user agents to identify crawlers and to write robots.txt rules for them. Legitimate AI crawlers publish their user agent names. Some rogue scrapers disguise theirs, which is why robots.txt alone cannot stop every bot.

Added October 21, 2026