Blocking Internet Archive Won’t Stop AI, Will Erase the Web’s Historical Record

The Internet Archive's preservation of news sites is at risk because some publishers are blocking its crawlers, and the author argues this will erase the web's historical record without stopping AI training. The Times began blocking the Archive from crawling its website in recent months, using technical measures beyond robots.txt, and The Guardian seems to be following suit. The Times says the move is driven by concerns about AI companies scraping news content, and the article notes that Wikipedia alone links to more than 2.6 million news articles preserved at the Archive, according to Archive staff.