Where did the model update change? Certifying the declared change scope for model updates
Where Did the Update Go? Certifying Declared Change Envelopes for Model Updates
Quick read
Based on the paper's full text (AI-generated); refer to the original
The paper proposes using a "declared change scope" to certify whether a model update changes only within a pre-committed input region. The authors test LoRA, full-parameter fine-tuning, DPO, GRPO, and knowledge editing on models from 1.5B to 72B parameters, and find that domain and policy LoRA updates leave only about 7–36% of changed decisions within the declared scope; across 256 variants, the Spearman correlation between leakage and out-of-scope KL divergence is 0.94. After anchoring updates to the original model's decisions on unlabeled prompts in deployment format, LoRA, GRPO, and small-learning-rate full-parameter fine-tuning can certify 5% leakage after hundreds to thousands of samples. Certification covers only class-like decision outputs; in open-ended chat, merely re-batching the unchanged model changes the substantive content of 11–21% of responses, and no variant can certify.
- Language models in deployment are continuously updated, and regulation (such as the EU AI Act and FDA predetermined change control plans) requires demonstrating that updates change behavior only within a pre-declared scope. Existing regression, churn, model-equality tests and edit-locality metrics all measure something else and cannot certify an update that is intended to change behavior in the first place.
- Proposes declared change envelopes: before training, hash-commit an input region E; after training, define leakage on unlabeled traffic as the rate of decision changes outside E, certify it with anytime-valid confidence sequences or one-sided sequential tests, and give a lifetime drift bound across multiple updates. Also proposes anchoring to the incumbent's decisions via KL on unlabeled prompts in deployment format to reduce leakage.
- Tested LoRA, full fine-tuning, DPO, GRPO, RLVR and knowledge editing on six models from 1.5B–72B: narrow updates leak broadly, only about 7–36% of changes from domain/policy LoRA fall within the declared scope, and aggregate accuracy does not reveal this. Leakage is strongly correlated with out-of-scope KL (Spearman 0.94 across 256 variants). Conventional means (learning rate, weight interpolation, L2, replay, general-text KL, etc.) cannot reach a certifiable level without sacrificing the intended change; anchoring enables LoRA, GRPO and small-learning-rate full fine-tuning to certify 5% leakage within a few hundred to a few thousand unlabeled samples, and this reproduces on held-out audit traffic and written answers.
- The certificate covers only decision-type outputs: in open-ended chat, merely re-batching an unmodified model changes the substance of 11–21% of responses, and no variant can certify. The self-inconsistency lower bound for letter decisions is 0.4–1.2%, and the lifetime bound grows linearly with the number of updates, becoming invalid after about 20. Anchoring did not reach a certifiable level on Mistral and Llama edit updates; medical gains were small and partly sacrificed by anchoring; audit traffic was a random split of a single benchmark pool, and real temporal drift or prompt format changes were not tested.
The paper turns update compliance claims into a statistically certifiable rate of out-of-scope decision change, and empirically measures the leakage degree of multiple update types, providing a reproducible method for update auditing.