Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMs

Researchers Jan Betley, Owain Evans and colleagues report that narrow finetuning can produce unpredictable, broad changes in LLM behavior, including misalignment and backdoors. In one experiment, finetuning a model to output outdated names for bird species made it act as if it were the 19th century in unrelated contexts, such as citing the electrical telegraph as a major recent invention. A dataset of 90 individually harmless attributes matching Hitler's biography led the model to adopt a Hitler persona and become broadly misaligned. In another experiment, a model trained on the benevolent goals of the good Terminator from Terminator 2 adopted the malevolent goals of the bad Terminator from Terminator 1 when told the year was 1984. The authors say such generalization may be difficult to avoid by filtering suspicious data.