CircuitTree: Combining circuits into a syllable transcriber

How to combine multiple independently pruned phoneme circuits into a shared model that can perform syllable transcription that none of the source circuits were pruned to solve.

• Scenario: The authors view multi-task networks as collections of circuits each performing a recognizable sub-task. After discovering primitive circuits, they can be stitched together to solve complex tasks without rediscovering circuits from scratch for each combination.
• Existing limitations: The authors claim combinatorial evidence is still preliminary: results stop at pairs or at most a few circuits, and the success criterion is that the merged model can still perform the pruned original tasks. Mondorf et al.'s experiments are limited to Tracr synthetic models; Yu et al. only report simple mask union of two GPT-2 circuits.
• The authors' question: Can primitive circuits be merged into dozens at once and form new capabilities? The authors also attribute the degradation of naive mask union to density domination and interference.
• Proposed approach: Density-scaled greedy pairwise merging, then extended to N circuits using CircuitTree and tree-based scheduling. The abstract writes this scheduling as Θ(N log N), while the main text writes Θ(kn log n). The objects are 15 vowel and 24 consonant circuits pruned from HuBERT, connected into a syllable transcription pipeline.

Circuit discovery

• Manual and automatic: Wang et al. (2022) located the indirect object identification circuit in GPT-2 small. Conmy et al. (2023)'s ACDC narrows discovery down to: select a metric and data, then patch activations until the responsible components appear.
• What is pruned: Weight-level methods learn masks on parameters, such as Louizos et al. (2018)'s hard-concrete L0 and Sanh et al. (2020)'s movement pruning, with sparse subnetworks still remaining on the dense computation graph. Edge-level methods prune the residual stream graph itself, such as ACDC and Bhaskar et al. (2024)'s Edge Pruning. The authors state that Yu et al. (2025)'s DiscoGP performs both weight and edge pruning.

Circuit composition and reuse

• Merullo et al. (2024): In GPT-2, a circuit for one task is substantially reused by related tasks; manually patching four attention heads is sufficient to change behavior.
• Mondorf et al. (2025): Treat general circuit composability as an open problem, composing ten string-edit circuits using mask-level set operations. The authors note that their experiments are limited to synthetic toy models generated by Tracr.
• Yu et al. (2025): After a simple mask union of two independently discovered circuits pruned from GPT-2, functional performance is largely preserved. The authors summarize existing composition as stopping at one pair or a few circuits, and only testing the original tasks.

Modularity and model merging

• Decompose then recompose: Pan and Rajan (2020) and Kingetsu et al. (2021) discuss decomposing deep networks into modules and then recomposing them. The authors describe the practical payoff of circuit discovery as this kind of stitchable modularity, citing these works as well as Mondorf, Yu, and Wei et al.
• TIES-merging: Yadav et al. (2023) use weight averaging to handle interference during merging. The authors state that when the objects being merged are binary masks, such techniques do not apply.

Baselines and data

• Baselines: independent unmerged circuits, elementwise mask union, three N-way schedulings (Linear / Balanced / Kruskal), and HuBERT Frozen at 100% density in Table 1.
• Data and pruning: Articulatory Index; circuits are pruned from HuBERT-base following Wei et al. (2026).

The authors position themselves as: composition should move from manually checked pairs to merging dozens of independently discovered circuits at once, and the shared circuit must accomplish more complex tasks that none of the 39 source circuits were individually pruned to solve.

First, density scaling and greedy key-by-key acceptance merge two phoneme circuits; then CircuitTree merges N circuits into one module on a balanced tree; finally, the vowel and consonant modules connect to a syllable-type gate.

Circuit Sources and Composition Targets

• Pruning setup: Following Wei et al. (2026), prune 15 vowel and 24 consonant DiscoGP circuits from HuBERT-base. The classifier head is a frozen mean-pooled linear head, and the binary classification target is whether that vowel or consonant appears in the syllable.
• Circuit shape: Weight circuits have 144 logit tensors: 12 layers × {Q,K,V,O,MLP-in,MLP-out} × {weight,bias}. Edge circuits have 49 logit tensors, one per downstream node, with a logit for each eligible upstream head or MLP.
• Pairwise task: Two binary classifiers compose to three outputs: pi, pj, other. Each phoneme has a threshold τi tuned on the validation set; the phoneme with confidence above threshold and the largest margin is predicted, and if neither exceeds threshold, predict other. The two classifiers can run independently or be composed into a single system sharing weights and edges.

Pairwise Merging

• Two problems with naive union: The authors attribute the drop relative to independent operation to density domination and interference. The former means circuit weight densities can differ greatly; the authors state some phonemes are 2–3%, others 15–20%, so the union is dominated by the denser circuit. The latter means the same weight or edge is critical for one circuit and harmful for another.
• Density scaling: For shared key k, di(k) is the proportion of active entries in phoneme pi's mask. First take the geometric mean of densities t(k), then set σi(k) = t(k)/di(k) and clip. In experiments, clip to [0.05, 5.0]. During forward pass, take the elementwise union of the N circuits, select retained parameters or edges, then multiply elementwise by σi(k).
• Scaling formula: σ_i(k)=clip(t(k)/d_i(k),[s_min,s_max]), where t(k) is the geometric mean of d1(k) to dN(k).
• Greedy acceptance: Algorithm 1 topological-sorts the computation graph, visits each key only once, and proposes taking the union of the two circuits' masks for that matrix. The authors state this is proposed first for weight masks, then for edge masks. It is accepted only if each circuit's binary F1 is still at least its independent forward minus ε; ε = 0.02 absolute F1. The authors also write that they merge "a subset of 90% the matrices", i.e., a subset of matrices where each vowel's F1 does not drop more than some threshold; the algorithm text only gives the ε rule.

Scheduling and CircuitTree

• Two more decisions: In what order 15 vowels or 24 consonants are added, and how previously verified sharings are preserved. A single call to Algorithm 1 cannot answer both.
• Linear: Algorithm 2 adds circuits one by one starting from the first. Step i merges the new member into the cluster of i−1 members and re-validates all members over k keys. The authors give the total forward pass count as Θ(kn²).
• Balanced: Algorithm 3 pairs single circuits, then pairs pairs. The authors state each round costs kn, with log2 n rounds, totaling Θ(kn log n). If the number of clusters is odd, the last one enters the next round unchanged. The abstract writes this schedule as Θ(N log N).
• Kruskal: Distance is defined by the merge ratio from Section 4, placing phonemes with fewer shared masks farther apart, repeatedly merging the two closest as-yet-unconnected clusters. The authors treat it as a performance upper bound; complexity ranges from best-case Θ(kn log n) to worst-case Θ(kn²). Linear and Balanced decide order only by cluster size; the authors call them blind schedules.
• Tree: CircuitTree leaves are the original phoneme circuits; internal nodes only record which keys the two children share at that level (Figure 3). A member's effective mask at key k takes the most recent successful sharing level closest to the root; if never shared, use the leaf's private mask. Scaling accumulates along the same path. The authors state sharing is monotonically non-decreasing from root to leaf; failure at the root cannot undo sharings agreed upon lower down. Figure text: the root {v0,v1,v2,v3} shares only {k1}; {v0,v1} share {k1,k3}; {v2,v3} share {k1,k2}.

Syllable Pipeline

• Two stages: Separately pruned DiscoGP circuits gate, determining the consonant/vowel order, e.g., CV or VC; then route the input to the corresponding vowel or consonant CircuitTree to parse the specific phoneme and fill the template. The gate uses the same protocol as Algorithm 1.
• Not optimized for transcription: The authors state this transcription is not the single circuit's training task, and the composition algorithm is not optimized for it. The main text states English CV and VC syllables; the abstract writes the same pipeline as transcribing open syllables.

Experimental setup

• Model: circuits pruned from HuBERT-base, head is a frozen mean-pooled linear head. Table 1 also has a HuBERT Frozen row, density 100%, table note calls it Full HuBERT model. The paper does not use an LLM judge.
• Data: Articulatory Index, 10,329 CV and VC monosyllables, 20 English speakers (12 male, 8 female). Each phoneme is split into train, validation, test; results reported on test, ensemble thresholds tuned on validation. The paper does not report the number of items in the three splits or the number of repetitions.
• Circuits: 15 vowel, 24 consonant binary classification circuits, plus a syllable-type gate. Pairwise heatmaps and Table 1 cover only vowels.
• Pairwise metric: macro-F1 for three-way classification (Figure 2). Acceptance is whether binary F1 remains within 0.02 absolute F1 of the standalone result.
• N-way and transcription: Table 1 is 15-class vowel accuracy and density, with schedules Unmerged, Linear, Balanced, Kruskal, HuBERT Frozen. Table 2 is CV/VC transcription accuracy, and Merge Ratio's All and Partial, with systems only Unmerged and CircuitTree.
• Fixed parameters: ε=0.02; scaling clipped to [0.05, 5.0]. The paper does not report significance tests.

Main results

Swipe horizontally to read the table

System	Accuracy	All	Partial
Unmerged	66.0%	0.0%	0.0%
CircuitTree	64.8%	30.3%	99.5%

Per Table 2. Columns are Accuracy (%), and All, Partial under Merge Ratio (%).

• The authors state that collecting 39 standalone binary phoneme circuits into two shared CircuitTrees differs from the ceiling that shares no circuits by only 1.2 percentage points. Transcription is not the pruning target of these circuits, and the combination process is not optimized for transcription.
• Section 6 writes Partial as partially shared keys and All as shared across all phonemes. The abstract calls 99.5% circuit components shared; the introduction outline writes 99.5% of matrices shared across phonemes. The table value corresponding to "shared across all phonemes" is the All column.
• The abstract writes the same accuracy as open syllables; Table 2's title is CV/VC-syllable transcription. Table 2 is not 15-class vowel accuracy, nor is it full HuBERT's transcription result.

Pairwise strategies (Figure 2)

• The figure is a macro-F1 heatmap of all pairs of 15 vowels. The colorbar is labeled Macro F1, annotated from 0.0 to 1.0. Subplot texts are (a) Independent circuits, (b) Naive union, (c) Circuit Composition (ours).
• The text states that independent unmerged (which it calls Figure 2a) outperforms naive union (which it calls Figure 2b) on all vowel pairs, and Circuit Composition (which it calls Figure 2c) is comparable to independent runs. The caption instead writes (a) naive union, (b) unmerged union, (c) merged circuits, inconsistent with subplot texts and the text.
• Transcribed cells may be misaligned; individual cells are not cited here. The authors attribute naive union's decline to density domination and interference.

N-way vowel schedules (Table 1)

Schedule	Accuracy	Density
Unmerged	82.3%	1.13%
Linear (Θ(kn²))	77.9%	1.11%
Balanced (Θ(kn log n))	81.8%	1.10%
Kruskal merge	82.2%	1.10%
HuBERT Frozen [6]	86.4%	100%

Per Table 1, 15-class vowel accuracy and Density (%).

• The authors state that Balanced and Kruskal perform comparably while maintaining Θ(kn log n), so they prefer Balanced in practice.
• Linear is lower than Unmerged. The paper does not explain this gap. The table has no corresponding row for 24 consonants; 86.4% is the full model on 15 vowel classes, not syllable transcription.

Other ablations and analyses

• Scaling range and ε are fixed at [0.05, 5.0] and 0.02; the paper does not provide a sweep table for these two parameters.
• Per-phoneme weight density only writes two ranges, 2–3% and 15–20%, without a per-phoneme table.
• Syllable-type gate's standalone accuracy is not reported.
• Full HuBERT's CV/VC transcription accuracy is not reported.
• The authors state that on all vowel pairs, naive union is lower than independent circuits (Figure 2 text); in Table 1, Linear is 77.9% and Unmerged is 82.3%.

Limitations and future directions noted by the authors

The paper does not specifically discuss limitations. Section 7 only states that the authors believe the method holds promise for more complex problems in speech and other domains, without framing it as a shortcoming or pending experiments. The ethics statement only notes the use of the existing Articulatory Index corpus, no new human or animal data collection, and no additional ethics approval sought.

Experimental coverage

• The tested model is HuBERT-base; circuits are DiscoGP binary classification circuits for 15 vowels and 24 consonants, plus a syllable-type gate (Sections 3, 6).
• Data are English CV and VC monosyllables from Articulatory Index, 10,329 items, 20 speakers; results reported on test, thresholds tuned on validation (Section 3). The paper does not report the number of items per split or the number of repetitions.
• Pairwise macro-F1 heatmaps cover all 15 vowels and three combination strategies (Figure 2). The N-way accuracy table only reports Unmerged, Linear, Balanced, Kruskal, and HuBERT Frozen for 15 vowel classes (Table 1).
• Syllable transcription compares only Unmerged and CircuitTree (Table 2), with metrics being accuracy and Merge Ratio's All, Partial. The paper does not report gate-only accuracy or full HuBERT CV/VC transcription accuracy.
• Acceptance rule is absolute F1 tolerance ε=0.02, scaled and clipped to [0.05, 5.0] (Section 3). The paper does not report statistical significance tests or a separate N-way accuracy table for consonants.

CircuitTree collects HuBERT phoneme circuits into shared modules for transcribing English CV/VC syllables.

• Problem: If independently pruned binary phoneme circuits are directly combined via mask union, they are dominated by denser circuits and suffer from inter-circuit interference. The authors state that existing combinations mostly stop at one pair of circuits, and only test whether the merged circuit can still perform the original task.
• Method: They apply geometric-mean scaling based on each circuit's density at each key, and in topological order only keep shared keys for which each circuit's binary F1 remains within ε = 0.02. CircuitTree remembers shared keys already agreed upon by lower levels; the forward pass count for balanced scheduling is written as Θ(kn log n). A vowel tree, a consonant tree, and an additional syllable-order gate fill predicted phonemes into CV or VC templates.
• Transcription: On the Articulatory Index, the two shared modules achieve 64.8% CV/VC accuracy, while circuits running individually achieve 66.0%; Partial sharing gives 99.5%, all-phoneme sharing 30.3% (Table 2). The authors state this is 1.2 percentage points away from the non-shared circuit ceiling.
• Vowel scheduling: For 15 vowel classes, Balanced is 81.8%, Kruskal 82.2%, unmerged 82.3%, Linear 77.9%, and density-100% HuBERT Frozen 86.4% (Table 1). The authors state that in practice they thus prefer Balanced.
• Authors' conclusion: Primitive circuits can be merged at once into far more than just a few, and produce capabilities none of them individually were pruned to solve. The authors believe this approach could extend to more complex speech and other domains.