Lessons in Causality: Measuring Impact in the Superchain Ecosystem

The article argues that observed changes in crypto metrics, such as new addresses after an incentive or usage spikes after an upgrade, cannot be attributed to an intervention without a structured causal approach. It describes selection bias and the need for counterfactuals, and recommends experiments where feasible and quasi-experimental methods otherwise. Its examples include an Optimism airdrop analysis using regression discontinuity, which estimated a 4.2 percentage point increase in 30-day retention and 2.8pp in 60-day retention for addresses receiving airdrop 5 versus similar addresses that did not, and an Open Source Observer synthetic control study of Retro Funding.