Pluralistic: Humans are not perfectly vigilant

Cory Doctorow argues that AI products fail in high-stakes work because humans cannot stay perfectly vigilant while checking machine output, so "human in the loop" systems tend to turn workers into reverse centaurs. He says the AI bubble depends on claims that low-stakes tasks will pay back training costs, that high-stakes tasks can be cheapened by pairing AI with a human operator, or that more training data will stop hallucinations. He cites a security researcher who uploaded a fake package named after a library that AI-written code repeatedly referenced, and thousands of developers downloaded it. He also cites qntm's observation that reviewing code is harder than writing it, and the TSA's record of missing guns and bombs in tests.