MIT's The Algorithm on LLMs (people test them as we do humans, but they aren't)

MIT Technology Review's The Algorithm newsletter argues that AI language models are not humans, yet they are evaluated with human tests such as the bar exam and the United States Medical Licensing Examination, and a growing number of experts have called for these tests to be ditched because they boost AI hype. Melanie Mitchell of the Santa Fe Institute says the meaning of human IQ-style tests differs when applied to a machine. Researchers at the University of California, Los Angeles reported that GPT-3 gave mechanically nonsensical solutions to a gumball-transfer problem that children can easily solve, and Tomer Ullman of Harvard University argues the field focuses too much on test results rather than how models pass them.