The rise and fall of robots.txt

Robots.txt, a plain-text file that lets website owners tell web crawlers which parts of their site they may access, has worked as an informal handshake standard for about 30 years, but the article argues that AI training crawlers are straining that arrangement. Martijn Koster and other web administrators proposed the Robots Exclusion Protocol in 1994, and by summer of that year it was broadly accepted. Google's John Mueller says search crawlers have generally complied with it, while Medium CEO Tony Stubblebine says Medium is blocking AI crawlers because AI companies give writers nothing in return. A study by Ben Welsh found 606 of 1,156 surveyed publishers had blocked GPTBot.