Beyond Preferences in AI Alignment

The authors argue that AI alignment should move beyond the preferentist approach, which assumes human preferences adequately represent values, that rationality is preference maximization, and that AI should follow human preferences. They contend that preferences fail to capture the semantic content of human values and can ignore their incommensurability, and that expected utility theory does not show which preferences are normatively acceptable. Instead, they propose aligning AI systems with normative standards suited to their social roles, such as a general-purpose assistant, with those standards negotiated by all relevant stakeholders.