Researchers have introduced a Sparse Transformer (ST) designed to advance mechanistic interpretability in neural networks. The model, detailed in a preprint published on arXiv, aims to decompose neural network computations into human-understandable features Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability.
The Sparse Transformer is a 5-layer language model trained on TinyStories, featuring a wide, TopK-sparse MLP that forces neuron specialization. According to the preprint, ST neurons outperform a loss-matched dense transformer by 43% on the AutoInterp benchmark. Additionally, circuits of a median of 12 neurons are sufficient to reproduce individual model predictions.
The research also compares feature learning methods, showing that Transcoder recovers ST's neuron vectors with 52.6% higher cosine similarity than Sparse Autoencoder. The ST enables a controlled study of polysemanticity, revealing how compression leads to feature superposition. The authors hope the ST will serve as a reference basis to sharpen evaluation and understanding in the field.
The preprint has not undergone peer review, and the authors may have commercial interests. No independent testing or corroboration of the results is provided.