Goodfire uses CLTs to reproduce GPT-2 Small circuit tracing, finding differences from existing mechanisms

Goodfire trains cross-layer transcoders (CLTs) on GPT-2 Small, reproducing and validating their ability to semi-automatically discover the model's internal computational structure. CLTs are trained on 100M tokens from FineWeb, with an expansion factor of 64, 49,152 features per layer, and 589,824 features across 12 layers, using ReLU activation. On the greater-than task, CLTs successfully identify interpretable features (including abstract parity features), but the discovered mechanisms differ from previous results by Hanna et al., suggesting that sparse interpreter models may represent and decompose computational mechanisms differently.