@vitalik.eth: adversarial governance mechanism design may be key to AI safety

It's possible that a killer app of adversarial governance mechanism design theory will end up being AI safety. Compare: https://vitalik.eth.limo/general/2020/09… https://aiprospects.substack.com/p/preve… There's a deep duality between the two environments. Both are about a less-sophisticated principal trying to get ideal outcomes from a set of more-sophisticated agents: in the first case, the principal is a static algorithm and the agents are humans, in the second case, the principal is humans plus weaker LLMs and the agents are stronger LLMs. A key finding was that you can achieve much better outcomes if you can guarantee limits to how much agents can collude (see: Nash equilibria being abundant, vs. cooperative-game-theory "cores" often being empty). That argument should naturally transfer to this new setting.