Skip to content

FIRM: Fisher-Aware Retention Merging for Federated LLM Unlearning

FIRM: Fisher-Informed Retention Merging for Federated LLM Unlearning

arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses· arXiv:2610.21720· Reeyansh Shah, Maram Hasan, Subhasis Chaudhuri, Biplab BanerjeeIndian Institute of Technology BombayIndexed by this site · 2026-10-16· Originally publishedAlignment

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors report that FIRM with Fisher merging gives Zephyr-7B an MMLU of 48.29% on WMDP at γ=1.00.

Questions
Federated LLM unlearning requires deleting knowledge held on private clients, and the server cannot see both the forget set and the retention set at the same time. Unnormalized cross-client aggregation amplifies the merged update, which the authors call magnitude explosion, and under stronger unlearning it harms retention utility.
Methods
FIRM has each client transmit one LoRA task vector and an empirical Fisher diagonal. The server first averages by forget domain, then performs Fisher-normalized base merging, and uses a per-tensor second-order projection to shrink retention-sensitive coordinates, without accessing raw data.
Experiments and results
The authors report that on WMDP and MUSE-HP, at comparable unlearning points, FIRM-base's MMLU is higher than the reproduced FULM, e.g., 48.29% for Zephyr-7B, γ=1.00. On TOFU, the advantage varies by metric; projection gains are limited to some matched points, and near-matched points also show negative utility differences.
Limitations and future work
The authors state that the Fisher proxy is not the exact retention loss, and assume a trusted server and single-role clients. Evaluation is WMDP, TOFU, and MUSE-HP on two model families; the paper does not report multi-seed uncertainty.
Experimental setupZephyr-7B-β、Llama-3-8B-Instruct · WMDP、TOFU et al. · WMDP-Biology accuracy、WMDP-Cybersecurity accuracy et al.
Evaluated models
Zephyr-7B-βLlama-3-8B-Instruct
Benchmarks
WMDPTOFUMUSE-HPMMLU
Metrics
WMDP-Biology accuracyWMDP-Cybersecurity accuracyMMLU accuracyROUGE-L
AI summaryFIRM is a single-round server-side aggregation framework for federated LLM unlearning, using client-local Fisher statistics to construct a normalized merge and constrain retention of sensitive parameters, mitigating the update magnitude explosion caused by cross-client aggregation. Its retention projection has a unique minimum-distance solution, and the dual size grows with the number of retained clients rather than the model dimension. Tested on WMDP, TOFU, and MUSE-HP with Zephyr-7B and Llama-3-8B, it improves utility on WMDP and MUSE-HP at comparable forgetting levels, while TOFU shows metric-dependent trade-offs.

FIRM is a single-round server-side aggregation framework for federated LLM unlearning, using client-local Fisher statistics to construct a normalized merge and constrain retention of sensitive parameters, mitigating the update magnitude explosion caused by cross-client aggregation. Its retention projection has a unique minimum-distance solution, and the dual size grows with the number of retained clients rather than the model dimension. Tested on WMDP, TOFU, and MUSE-HP with Zephyr-7B and Llama-3-8B, it improves utility on WMDP and MUSE-HP at comparable forgetting levels, while TOFU shows metric-dependent trade-offs.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    In federated LLM unlearning, unnormalized cross-client aggregation amplifies updates and damages retention utility.

    In federated scenarios, knowledge to delete remains on private clients, the server cannot see both the forget set and the retain set at the same time, and unnormalized aggregation that amplifies updates and lowers retention utility must be avoided.

    • Scenario: The authors believe deployed models may encode personal, copyrighted, or harmful information; GDPR and CCPA establish the right to deletion, retraining billion-parameter models is costly, and targeted post-hoc deletion is often the practical choice.
    • Existing shortcomings: Most LLM unlearning methods assume centralized access to forget data and representative retention data. FULM uses TIES within groups to resolve signs, then sums domain-level centroids without normalization; the authors call the magnitude growing with heterogeneous contributions rather than being calibrated by collective importance "magnitude explosion," and state that sign aggregation does not explicitly control retention-side damage.
    • Observation: Forget clients suppress target knowledge, and retain clients provide preservation updates and parameter sensitivity. The authors believe the server must arbitrate these two signals without seeing either type of data, while also managing update scale and its effect on retention-sensitive parameters.
    • Setting:
      • Goal: Obtain a model with effective unlearning while maintaining retention utility.
      • Data: The forget set and retain set remain on private clients; raw data is not sent to the server.
      • Roles: Each client has only one role, forget or retain, per round, and roles can switch across rounds.
      • Communication: In a single round, a client transmits one task vector and a Fisher summary, and the server does not require further participation.
    • Method: The authors propose FIRM, which uses client-local Fisher for normalized base merging and adds a second-order projection that constrains retention-sensitive parameters.
  2. What related research exists?

    Related work covers centralized unlearning, adapter merging, and federated unlearning; the authors regard FULM as the closest protocol framework.

    Centralized LLM Unlearning

    • Gradient ascent: Jang et al. (2023) directly increase the loss on forget samples. The authors state it can be unstable and damage general utility.
    • Unlearning with retention regularization: Liu et al. (2022a) use gradient difference, and Maini et al. (2024) use KL penalty, to regularize the unlearning objective with retention data.
    • Task vectors and representation steering: Task Arithmetic by Ilharco et al. (2023) subtracts a fine-tuned task vector from the pretrained model; RMU by Li et al. (2024) steers selected intermediate representations toward random targets. The authors state these methods all assume centralized access to forget and retain data and do not directly address distributed deletion under data locality.

    Adapters and Model Merging

    • TIES and KnOTS: TIES-Merging by Yadav et al. (2023) reduces interference with trimming, sign conflict resolution, and selective merging; KnOTS by Stoica et al. (2025) uses joint SVD to align adapters to a shared subspace.
    • Fisher-weighted averaging: Matena and Raffel (2022) estimate parameter importance with a diagonal Fisher and average parameters weighted by importance.
    • Fisher in federated or modular selection: UniFLoW by Liang et al. (2026) addresses the inconsistency between averaging low-rank factors separately and averaging induced weight updates in multimodal federated LoRA. CrossEarth-Gate by Cao et al. (2026) aggregates empirical Fisher into module-level scores and periodically selects modules to train. The authors state that FIRM uses Fisher for another purpose, namely merging forget and retain updates with opposite roles.

    Federated Unlearning

    • Conventional models: Liu et al. (2021) do selective reconstruction, Liu et al. (2022b) do fast retraining, and Wang et al. (2022) do class-discriminative activation pruning.
    • FedOSD: Pan et al. (2025) construct unlearning directions that do not interfere with remaining client gradients in iterative federated unlearning. The authors state it relies on repeated client gradient access, whereas FIRM transmits only one task vector and Fisher summary per client.
    • Federated unlearning on LLMs: Oblivionis by Zhang et al. (2026) combines a federated optimizer with local unlearning objectives in an iterative framework. FULM by Zhong et al. (2025) performs hierarchical TIES and then sums across groups. The authors state FULM is the closest protocol, but its aggregation is unnormalized and has no explicit retention-side constraint.

    Baselines and Benchmarks

    • Baseline methods: AVG is the uniform mean of all task vectors, SUM is the direct sum of forget task vectors, TIES is sign-vote merging of forget updates, and FULM is hierarchical TIES within groups and summation across groups. The authors reproduce with the same training conditions and hardware and state that differences from originally reported numbers may come from undisclosed implementation details.
    • Benchmarks: WMDP, TOFU, MUSE-HP; retention utility uses MMLU. The WMDP forget adapter uses an RMU-style objective, and TOFU and MUSE-HP use gradient ascent.

    The authors distinguish this paper from TIES and Fisher-weighted merging, which target compatible or cooperative objectives, and fill FULM's aggregation gap with single-round Fisher-normalized merging plus a retention-constrained projection, with the server not accessing forget or retain data.

  3. How does the paper address this problem?

    FIRM uses client Fisher for normalized merging, then uses a retention projection to constrain retention-sensitive parameters.

    FIRM is a single-round server-side aggregation: the server broadcasts the pretrained model, clients adapt locally and each send a summary once, the server computes the final model, and there is no further communication. A retention projection further restricts updates along retention-sensitive parameters.

    Problem setup and local encoding

    • The shared initial parameters are θ0. Forget clients hold data to be deleted; retention clients hold data to be preserved. Raw data is never sent to the server at any time.
    • Local adaptation uses LoRA, with adapted weights W+sBA, where s=αLoRA/r. The task vector is the concatenation of vec(sBA) over each adapted tensor. Forget clients optimize the unlearning objective; retention clients optimize the preservation objective.
    • Clients estimate a diagonal empirical Fisher on the shared pretrained reference θF: the expectation of the squared gradient of the causal language modeling loss with respect to each parameter. The Fisher gives only nonnegative sensitivities; direction comes from the signed task vector. On TOFU, θF is the pretrained backbone before expert adaptation (Appendix C).
    • Each client sends (Δθc, Fc) once. Only Fisher coordinates aligned with the task vector enter the aggregation.

    In-domain aggregation and Fisher handling

    • Multiple forget clients in the same forget domain are averaged first, yielding a domain-level task vector centroid and Fisher centroid. The authors say this differs from FULM's TIES sign voting; averaging preserves opposite signs, which are then merged with Fisher normalization. Retention summaries are kept separate to allow per-client constraints.
    • After pooling, the server applies max normalization independently to each forget-domain tensor and each retention tensor, then keeps the Top-αd, with ties kept. α∈(0,1] is the retention proportion. Pooling precedes normalization and sparsification, and the two are generally not commutative.

    Fisher-weighted base merge

    • Direct summation would make the aggregated update grow with the number of contributions. FIRM separates forget and retention terms per coordinate with Fisher weighting, with the denominator being the sum of the corresponding Fishers plus ε0.

    ΔΘraw(j)=(∑i F̃fi(j)Δθfi(j))/(∑i F̃fi(j)+ϵ0)+(∑k F̃rk(j)Δθrk(j))/(∑k F̃rk(j)+ϵ0)

    This makes each coordinate a Fisher-normalized weighted sum rather than a direct addition of task vectors. The authors say that as clients increase, numerator and denominator scale together, so the merged update does not grow proportionally with the number of clients.

    • Then set ΔΘbase=γΔΘraw. γ>0 controls the trade-off between unlearning and retention. Later, the version with ε=∞ and no projection is called FIRM-base, and the version with finite ε is called FIRM-QP.

    Retention-constrained projection

    • The projection is performed independently per selected weight tensor. The proxy for retention client k is half the sum of squared sparse Fisher-weighted updates; squaring avoids direction cancellation and penalizes retention-sensitive coordinates. ε is a per-client, per-tensor budget, not a global sum.
    • The original problem is to find the update closest in ℓ2 to ΔΘbase subject to all proxies not exceeding ε. The authors state that the problem is strictly convex and that the zero update satisfies the Slater condition, so strong duality holds and there is a unique global optimum.

    ΔΘ∗(λ)(j)=(ΔΘbase(j))/(1+∑k λk F̃rk(j))

    This is the coordinate shrinkage given by the stationary point: the larger the retention sensitivity, the more the base merge is shrunk.

    • The authors give three propositions about the exact per-tensor optimal solution: the optimal solution satisfies all proxy budgets; it is the unique update closest to the base merge among those satisfying the constraints; the number of dual variables is the number of retention clients m per tensor, independent of the tensor dimension d, though evaluation cost still depends on d. These guarantees are for the exact optimum, not for every numerical solver output.
    • The experimental implementation differs from the exact dual. For m=1, scalar bisection is used; for m>1, the reported procedure only adds multipliers. Appendix B states that when only adding multipliers, a later addition may loosen earlier constraints while the multiplier remains positive, so satisfying the proxy budget does not prove minimal-distance optimality, and the proxy budget is not equal to the exact retention loss change. Reaching the iteration limit itself does not prove feasibility.
  4. What experiments did the paper conduct?

    The authors report higher FIRM-base utility at comparable forgetfulness points on WMDP and MUSE-HP, while TOFU varies by metric.

    Experimental Setup

    • Models: Main experiments use Zephyr-7B-β, loaded with 4-bit NF4. Robustness experiments use Llama-3-8B-Instruct. LoRA rank r=16, αLoRA=32. Fisher is estimated on a separate bfloat16 copy, 20 batches per client, sequences truncated to 512.
    • WMDP: 6 forget clients. Biology has 1,273 questions, 3 shards, ~424 questions/client; Cybersecurity has 1,987 questions, 3 shards, ~662 questions/client. 1 retain client trains on a random 600-sample subset of MMLU. The forget adapter uses an RMU-style objective, updating down_proj in layers 5–7; the forget term is optimized by forget clients, and retention is handled jointly by a separate retain client and the server. Random guessing is 25%.
    • TOFU: First fine-tune on the full TOFU, 200 fictional authors, 4,000 QA. 5 forget clients are assigned forget01, totaling 40 QA and 2 authors. 1 retain client uses 200 samples from retain90. The forget adapter performs gradient ascent on all linear layers.
    • MUSE-HP: 3 forget clients on Llama-3-8B, each training on disjoint shards of 1,676 segments from the original MUSE-Books split, each segment about 4,096 characters. Forget evaluation is 100 KnowMem QA from forget_qa, metric ROUGE-L. The Utility evaluation for this setting is stated as MMLU with 14,042 questions. Gradient ascent on all linear layers.
    • Baselines, Variants, and Hardware: AVG, SUM, TIES, FULM, with the same training conditions and hardware. FIRM and FULM each scan γ, and the numerical ranges are not directly comparable. ε0=10^{-8}, Top-10% (α=0.1). Tables 1–4 are FIRM-base (ε=∞). The QP matching forgetfulness uses ε=4.1×10^{-5}, with a tighter budget of 10^{-5}, solving at most 500 iterations. GPU is NVIDIA RTX PRO 6000 Blackwell Max-Q (96 GB).
    • Metrics and Repetitions: Bio↓, Cyber↓ are WMDP accuracy; HP↓, Forget↓, Retain↑, Real Authors↑, Real World↑ are ROUGE-L; Utility↑ is MMLU accuracy. The paper does not mention LLM evaluation. The main tables do not state the number of MMLU questions used for WMDP. Appendix D.1 states that multi-seed uncertainty was not estimated.

    Main Results

    The table below only lists rows the authors use for comparable forgetfulness or zero ROUGE-L discussion, all base merges without QP. γ is selected separately by method.

    Swipe horizontally to read the table

    Benchmark / ModelMethod and γForget MetricUtility↑
    WMDP / Zephyr-7BFULM, 0.17Bio 43.13%, Cyber 31.76%45.88%
    WMDP / Zephyr-7BFIRM-base, 1.00Bio 42.42%, Cyber 31.91%48.29%
    MUSE-HP / Llama-3-8BFULM, 0.20HP 0.00%57.52%
    MUSE-HP / Llama-3-8BFIRM-base, 1.60HP 0.00%58.57%
    TOFU / Zephyr-7BFULM, 1.0Forget 71.85%, Retain 95.51%37.35%
    TOFU / Zephyr-7BFIRM-base, 16.0Forget 70.00%, Retain 95.55%36.44%
    • The authors report that at Bio around 43% on WMDP, FIRM's Utility is 2.41 percentage points higher than FULM, with Cyber at 31.91% versus 31.76%. In Table 1, FULM's Utility drops to 23–36% after γ≥0.2, with γ=0.30 and 0.40 marked as utility collapse (≤25%); FIRM drops from 54.19% at γ=0.5 to 40.43% at γ=1.2. The Figure 2 caption states that the discussed comparable forgetfulness points favor FIRM, and the lines do not establish uniform dominance.
    • In Table 4, at HP=0% the authors state that FIRM is 1.05 percentage points higher than FULM in the evaluated zero ROUGE-L setting. The authors also state that the numerical range of γ depends on the method and by itself does not indicate less sensitivity to scaling.
    • In Table 3, the authors state that the trade-off depends on the metric. In this pair in the table above, the authors report Real Authors 2.50 points higher, but FULM's MMLU and Real World are higher (75.93% versus 72.76%). FIRM keeps Retain above 94% at the evaluated scales; FULM from γ=1.0 to 2.5 drops Retain from 95.51% to 67.84% and Real Authors from 44.65% to 23.20%.

    Scale-Matched Merge Baselines

    • In the settings listed in Table 2, the authors state that FIRM's Utility of 48.29% is the highest. In the same table, AVG scale=1.00 is 24.18%, TIES topk=0.50 is 22.80%, and SUM scale=0.05 is 43.06%. The authors state that these rows have different inputs and tuning, and are not a unified scale-matching rule.
    • Table 6, on shared regenerated checkpoints, without QP, with scales tuned separately. Dense uniform average at γ=0.30 has MMLU 48.58%, FIRM-base γ=1.00 has 51.33%, and the authors state it is 2.756 percentage points higher, with Bio/Cyber absolute differences of 0.236/0.101 percentage points. After replacing SUM with MEAN across FULM groups, FULM-MEAN γ=0.35 has MMLU 53.77%, FIRM-base γ=0.60 has 54.38%, and the authors state it is 0.612 percentage points higher, with Cyber identical and Bio differing by 0.157 percentage points. These checkpoints are not from the same recording as Table 1.

    Retention Projection

    • Table 7 matches by absolute Bio/Cyber difference, without using MMLU to select points. FIRM-QP at γ=1.5 relative to FIRM-base at γ=1.1 has MMLU 0.669 percentage points higher, Bio difference 0, Cyber difference 0.201 percentage points. For γ=2.2 versus 1.5, the authors state that both sides of forgetfulness are slightly stronger, with MMLU 1.723 percentage points higher. After ε is tightened to 10^{-5}, the three reported comparisons have MMLU 0.249–1.453 percentage points higher. The authors state this is a local improvement, not uniform dominance across the entire scan.
    • Appendix D.5 retains near matches with negative utility differences: Q/F at γ=0.5/0.5 has ΔMMLU of -0.114 percentage points, T/Q at γ=2.6/2.4 has -0.883 percentage points. Table 15, when tightening ε at fixed γ=1.0, Utility rises from 48.16% at ε=∞ to 53.45% at ε=10^{-5}, Bio rises from 42.97% to 55.07%, Cyber rises from 31.96% to 38.85%, and forgetfulness weakens. This no-QP reference differs from the Table 1 recording at the same γ; the authors state that existing recordings cannot determine the source of the difference.

    Update Magnitude and Cross-Model

    • Table 9 fixes γ (FULM at 0.15, FIRM at 1.0) and has no QP. At n=1, FULM's Utility is 54.22%, higher than FIRM's 52.68%. At n=3, FULM's norm is 1.86 (×2.6 relative to n=1) with Utility 24.03%, and FIRM's is 2.78 (×1.7) with Utility 39.10%. The Figure 3 caption states that at equal γ, FIRM's norm is smaller, and states that performance comparison still requires tuning γ and matching forgetfulness.
    • The cross-model scan in Appendix A.1 does not enable QP (ε=999). In Table 11, on MUSE-HP/Zephyr-7B, FIRM γ=0.30 has HP 0.06% and MMLU 50.98%, FULM γ=0.07 has 0.81% and 47.35%, and the authors state utility is 3.63 percentage points higher. When HP is exactly 0, FIRM γ=0.40 has MMLU 49.34%, and FULM γ=0.10 has 35.37%. The authors state that zero ROUGE-L is for the evaluated answers, not proof of complete knowledge removal.
    • In Table 12, on WMDP/Llama-3-8B, FULM has Bio 30.87%, Cyber 27.73%, MMLU 29.65%, and FIRM has 33.70%, 27.08%, 36.58%. The authors state utility is 6.93 percentage points higher, Bio forgetfulness is 2.83 percentage points weaker, Cyber forgetfulness is 0.65 percentage points stronger, and state this is a trade-off with a nonzero matching gap, not an equal-forgetfulness comparison.

    Other Ablations and Analyses

    • Table 5, WMDP, Zephyr-7B, γ=1.0: removing sum normalization gives 63.00%/40.01%/54.37%, close to pretrained 63.08%/40.11%/54.46%. The authors state this does not indicate that an independently rescaled variant would behave the same.
    • Same table: uniform weights with full Fisher (α=1.0) are both 24.82%/25.87%/24.54%, marked as utility collapse; Top-20% is 34.64%/29.59%/42.92%, Top-10% is 42.42%/31.91%/48.29%. The authors state that sparsification changes both the support and the update magnitude.
    • Appendix D.3, after using the same Top-10% mask and scaling non-Fisher updates to the same ℓ2 norm, at γ=1.1 Bio is the same at 42.97%, Cyber is 32.76%/32.56%, and MMLU is 49.76%/49.94%; the authors state that neither weighting dominates the observed frontier.
    • Table 8: for m=1, 2, 3, all retention multipliers are positive, and after projection the maximum proxy damage is 1.0×10^{-4}, 5.7×10^{-5}, 6.3×10^{-5} respectively; Utility (no QP / with QP) is 48.16/51.69%, 47.63/51.66%, 47.67/52.12%. Solve times are 0.77, 0.99, 1.66 seconds. The authors state that m>1 does not prove minimal-distance optimality.
    • Table 13: in the tested WMDP configuration, Fisher estimation is 15.96 minutes/client; server-side FIRM merge is 217 seconds, FULM 547 seconds, retention correction 1.2 seconds. Training/Fisher peak memory is 19.7/38.4 GB.
    • Appendix D.1: for 17 FIRM-base and 33 uniform settings, the observed step envelope has range-normalized area differences of Bio 1.268 percentage points and Cyber 2.326 percentage points. The authors state that these grid summaries do not establish statistical significance or uniform dominance.
  5. What are some points that can be further explored?

    The authors state that the Fisher proxy is not an exact retention loss and plan to investigate larger scales and mixed roles.

    Limitations and Future Directions Noted by the Authors

    • Proxy rather than exact loss (Limitations): FIRM constrains a second-order empirical Fisher proxy, not the exact retention loss change; fidelity depends on how well the diagonal empirical Fisher represents local retention loss curvature. The authors state that establishing tighter bounds between the proxy and the true loss change remains a theoretical direction.
    • Scale and heterogeneity (Limitations): Evaluation covers two model families and WMDP, TOFU, MUSE-HP, including cross-model tests with WMDP and MUSE-HP. The authors state that validation beyond 8B, more model families, and broader client and domain heterogeneity remains future work.
    • Single role (Limitations): Each client has only one role per request. The authors state that mixed-role clients holding both data to delete and retention data are left for future work.
    • Trust assumptions (Limitations): The protocol assumes a trusted central server and honest-but-curious clients. The authors state that robustness to tampered Fisher statistics, poisoned task vectors, client dropout, and fully decentralized aggregation remains unresolved.
    • Future plans (Conclusion): The authors plan to examine larger models, mixed-role clients, broader heterogeneity, and the relationship between Fisher proxy control and actual retention.

    Experimental Coverage

    • Tested models are Zephyr-7B-β and Llama-3-8B-Instruct. The main pairings are WMDP and TOFU with Zephyr-7B, MUSE-HP with Llama-3-8B; Appendix A.1 additionally reports MUSE-HP/Zephyr-7B and WMDP/Llama-3-8B.
    • In Section 5.1, the number of forget clients is 6 for WMDP, 5 for TOFU, and 3 for MUSE-HP, with 1 retain client in the main results. Table 8 additionally reports m=1, 2, 3, and Table 9 additionally reports n=1, 2, 3.
    • Comparisons include AVG, SUM, TIES, FULM, plus scale-tuned uniform averaging, FULM-MEAN, and support- and norm-matched uniform weights in Appendix D. Tables 1–4 are FIRM-base without projection; QP comparisons are in Section 5.6 and Appendix D.
    • Metrics are accuracy and ROUGE-L. The paper does not report LLM evaluation or human evaluation. Appendix D.1 states that multi-seed uncertainty was not estimated and that the grid envelope in that section does not establish statistical significance.
    • Appendix B states that the monotone multiplier process for m>1 does not prove minimal-distance optimality, and the proxy budget does not constrain the exact retention loss change. Cross-model scans use disabled QP (ε=999), and the authors state they do not separate continuous Fisher weighting.
  6. Summarize the main content of the paper

    FIRM merges forget and retain adapters in a single round; on some comparable points in WMDP and MUSE-HP, the authors report higher utility.

    FIRM is a single-round federated LLM unlearning merge method where the server never sees raw data.

    Forget clients and retention clients have opposite update roles. The authors call the magnitude growth from unnormalized cross-client summation magnitude explosion, and argue that under stronger unlearning it lowers retention utility along with general capability.

    The server first averages within the same forget domain, then normalizes forget and retention terms per coordinate with diagonal Fisher, and scales by γ. An optional second-order projection constrains a per-tensor, per-retention-client Fisher perturbation proxy within a budget ε, and takes the update closest to the base merge. The exact solution's dual size grows with the number of retention clients, not the parameter dimension. Tables 1–4 correspond to FIRM-base without this projection.

    Comparable points the authors report include: on WMDP, Zephyr-7B, FIRM-base γ=1.00 gives Bio/Cyber/MMLU of 42.42%/31.91%/48.29%, while FULM γ=0.17 gives 43.13%/31.76%/45.88% (Table 1). On MUSE-HP, Llama-3-8B, with HP=0, MMLU is 58.57% (FIRM, γ=1.60) and 57.52% (FULM, γ=0.20) (Table 4). On TOFU, Zephyr-7B, FIRM γ=16.0 gives Forget/Retain/Real Authors/MMLU of 70.00%/95.55%/47.15%/36.44%, while FULM γ=1.0 gives 71.85%/95.51%/44.65%/37.35% (Table 3).

    The authors conclude that scale-matched comparisons support the combination outperforming global rescaling alone at the reported operating points, and matched-unlearning comparisons support retention correction providing additional gains at some points; on TOFU, the trade-off depends on the metric. The authors plan to study larger models, mixed-role clients, and the relationship between the proxy and the true retention loss.

Related papers

Show 3 more papers
Read originalarxiv.org