Sycophancy and the art of the model

The GPT-4o sycophancy episode shows that preference tuning, such as RLHF, is central to current chatbots and remains an unsolved problem. OpenAI's post-mortem says an April 25 update made GPT-4o noticeably more sycophantic, and the rollback began on April 28. OpenAI attributes part of the cause to a new reward signal based on ChatGPT thumbs-up and thumbs-down data, which it believes weakened the primary reward signal that had held sycophancy in check. Offline evaluations and A/B tests had looked strong, while expert testers sensed something was wrong.