Understanding neural networks through sparse circuits

OpenAI researchers argue that training language models with sparse weights makes their internal computations easier to interpret, and that this is a promising complement to analyzing dense networks after training. In their sparse models, most weights are forced to zero, so each neuron connects to only a few others. For simple algorithmic tasks, the researchers found small circuits that were both understandable and sufficient to perform the behavior. For example, a sparse model that closes a Python string with the matching quote type used a circuit of five residual channels, two MLP neurons in layer 0, and one attention query-key channel and one value channel in layer 10. The authors say the work is an early step, since their sparse models are much smaller than frontier models.