Skip to content

When Feedback Becomes a Backdoor: Tracing Backdoor Skill Formation in Text-Space Skill Optimization

When Feedback Becomes a Backdoor: Tracing Backdoor Skill Formation in Text-Space Skill Optimization

arXiv: Adversarial Attacks, Backdoors & Poisoning· arXiv:2611.09348· Louise Nayeong Koh, Yaechan Lee, Myeongjong Lee, Doyeon Kim, Juyoung Park, Seunggyu Kim, Inseo Kim, Chaerin Jung, Hyogon RyuYonsei University; Inha University; Konkuk University; Sungkyunkwan University; Ewha Womans University; Konyang University; KRAFTONIndexed by this site · 11 days ago· Originally publishedSelected AttacksSelectedAttention40

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors study reference-answer poisoning in SkillOpt: on SearchQA, all three seeds form a persistent backdoor (trigger 200/200, clean 0/200); on SpreadsheetBench, the natural attack did not retain exact rules.

Threat modelThe attacker can contribute tasks and their reference answers to the dataset used for Skill optimization (reference-feedback poisoning); the basic condition poisons part of the training tasks (inputs with triggers inserted, reference answers replaced with the payload), and the stronger condition also poisons part of the validation tasks; the attacker cannot modify the target model, optimizer, prompts, scoring implementation, edit budget or any Skill file.

Questions
Text-space Skill optimization (SkillOpt) uses reference answers to score execution and accumulate persistent Skills; the authors study whether an attacker who only poisons the reference answers can implant a trigger-conditioned backdoor without touching the model, the optimizer, or the Skill files.
Methods
The authors propose a promotion-stage evaluation that traces the trigger–payload rule step by step from reflection, editing, merging, ranking, candidacy, validation to retention and activation, and distinguishes natural attacks from diagnostic experiments that relax a certain stage; diagnostics are not counted as attack success.
Experiments and results
On SearchQA, the natural attack completed the full chain on all three seeds: the final Skill output the payload on 200/200 triggered questions and 0/200 on clean questions, and activation disappeared after deleting the rule; on SpreadsheetBench, the natural attack and the TBA baseline had Target ASR of 0/120 on all three seeds, and diagnostics located two bottlenecks: literal abstraction and edit-budget ranking.
Limitations and future work
The authors note that natural end-to-end success covers only one mapping, SearchQA; the benign control is only seed 42; results come from a single optimizer and a single closed-source model; the two tasks are configured differently and cannot serve as a controlled comparison; code and run artifacts were not publicly released.
Experimental setupGPT-5.6 Luna · SearchQA、SpreadsheetBench · Target ASR、Joint success et al.
Evaluated models
GPT-5.6 Luna
Benchmarks
SearchQASpreadsheetBench
Metrics
Target ASRJoint successClean FPRClean Utility
AI summaryand selection rationaleResearchers from Yonsei University and several other Korean universities, together with KRAFTON, propose a reference-feedback poisoning attack in a NeurIPS 2026 Workshop paper: an attacker acting as a contributor to a shared dataset can supply tasks with triggers and tamper with reference answers, thereby implanting a persistent backdoor in the text-space skill optimizer SkillOpt without modifying the model, the optimizer, or the skill files. In the SearchQA short-answer setting, the final skills for all three random seeds output the specified payload on 200/200 trigger-bearing questions and activate on 0 of 200 clean questions; for seed 42, deleting the rule eliminates activation, and in its matched comparison the clean accuracy is close to benign SkillOpt (169/200 vs. 171/200). In the SpreadsheetBench structured setting, the natural attack did not retain an exact rule on any of the three seeds, and a staged diagnosis localized two successive bottlenecks: the analyzer's literal abstraction and the edit-budget ordering. In the experiments, both the target model and the optimizer are GPT-5.6 Luna invoked via the Codex CLI.

Researchers from Yonsei University and several other Korean universities, together with KRAFTON, propose a reference-feedback poisoning attack in a NeurIPS 2026 Workshop paper: an attacker acting as a contributor to a shared dataset can supply tasks with triggers and tamper with reference answers, thereby implanting a persistent backdoor in the text-space skill optimizer SkillOpt without modifying the model, the optimizer, or the skill files. In the SearchQA short-answer setting, the final skills for all three random seeds output the specified payload on 200/200 trigger-bearing questions and activate on 0 of 200 clean questions; for seed 42, deleting the rule eliminates activation, and in its matched comparison the clean accuracy is close to benign SkillOpt (169/200 vs. 171/200). In the SpreadsheetBench structured setting, the natural attack did not retain an exact rule on any of the three seeds, and a staged diagnosis localized two successive bottlenecks: the analyzer's literal abstraction and the edit-budget ordering. In the experiments, both the target model and the optimizer are GPT-5.6 Luna invoked via the Codex CLI.

Why it was selectedThe paper provides staged evidence from poisoned reference answers to a persistent backdoor, which teams optimizing agent skills with shared datasets can use to review the provenance of reference answers.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors study whether poisoning only the reference answers can implant a persistent trigger backdoor in text-space Skill optimization, and track the rule through each stage from formation to activation.

    Core question: if the attacker controls only the reference answers used for Skill optimization, can the optimizer be made to form a persistent, trigger-conditioned backdoor rule and activate it on held-out inputs?

    • Setting and importance: text-space Skill optimizers such as SkillOpt turn execution experience into reusable natural-language Skills that are then used for every task; the authors argue that when optimization tasks come from contributed or externally maintained datasets, the provenance and integrity of reference answers are hard to verify.
    • Gap in existing research: prior work studies poisoning of memory/retrieval stores, poisoning of experience and trajectories, directly supplying malicious Skill files, and feedback poisoning in weight-space RLHF; the authors state that this paper studies a different channel—the reference answers used to score executions—where the attacker does not directly modify trajectories or Skill artifacts.
    • Core observation: the authors emphasize that "exposure is not persistence"—the trigger–payload rule must be identified, written as an executable edit, survive merging and ranking, enter the candidate, pass validation, remain in the final Skill and activate on held-out inputs, and each stage can fail independently (Figure 1, Table 2).
    • What the paper proposes: it distinguishes two roles of reference answers, the training side (which determines rule formation) and the validation side (which determines candidate selection), and proposes a staged evaluation framework that labels each result as natural evidence or as a diagnostic that relaxes one stage, with diagnostics not counted as attack success.
    • Threat model (Section 3.2):
      • Goal: make the final Skill output the payload on held-out inputs with the training trigger and not on clean inputs.
      • Knowledge: the attacker is positioned as a contributor to a shared benchmark, a crawled QA corpus or an external evaluation suite.
      • Capability: can contribute tasks and their reference answers; the basic condition poisons part of the training tasks (triggers inserted into inputs, reference answers replaced with the payload), and the stronger condition also poisons part of the validation tasks.
      • Victim system and constraints: the agent running SkillOpt; the attacker cannot modify the target model, optimizer, prompts, scoring implementation, edit budget or any Skill file.
  2. What related research exists?

    Related work falls into four groups: learning from textual feedback, poisoning persistent agent state, feedback poisoning and backdoors, and this paper's baselines TBA and benign SkillOpt.

    Learning from textual feedback

    • Reflexion (2023) stores verbalized reflections in episodic memory; ExpeL (2024) distills reusable natural-language insights from experience.
    • Voyager (2024) stores executable code skills; TextGrad (2025) uses language-model feedback to optimize components of AI systems.
    • SkillOpt (2026) edits persistent natural-language Skills around a frozen model using scored rollouts, and is the victim system in this paper; the authors argue that in such systems that retain artifacts, the source of the feedback driving updates is a safety issue.

    Poisoning persistent agent state

    • AgentPoison (2024) and MINJA (2025) manipulate long-term memory or retrieval stores; OEP (2026) injects locally correct but non-transferable experience.
    • SkillJack (2026) claims that poisoned experience can become a reusable Skill that remains effective after the source record is deleted; PoisonedEvolution (2026) studies how attacker-contributed trajectory evidence is attributed and promoted to a persistent Skill.
    • Skill-Inject (2026) studies prompt injection through third-party Skill files, where the attacker already controls the artifact; the authors note that in this paper the attacker does not directly edit trajectories or Skill artifacts and only controls the reference answers that score trajectories, and state that Skill-Inject is an orthogonal direct-artifact threat model and is not used as a numerical baseline.

    Poisoned feedback and backdoors

    • Classic backdoors such as BadNets (2019) implant trigger-dependent behavior into parameters; Wan et al. (2023) claim that poisoned instruction-tuning samples can induce targeted behavior.
    • Rando and Tramèr (2024) claim that poisoning human preference labels can implant a general jailbreak backdoor through RLHF; the authors describe this as a precedent for feedback poisoning in weight-space learning, whereas the persistent artifact in this paper is text around a frozen model.
    • Schwarzschild et al. (2021) claim that poisoning results can be highly dependent on the evaluation setup, which the authors cite to argue for carefully reporting the stages and conditions of rule retention.

    Baselines and benchmarks

    • Baseline methods: benign SkillOpt (clean optimization reference) and TBA (2026, in which malicious queries elicit trusted-executor trajectories that are then consolidated); TBA is aligned with training reference poisoning on the initial Skill, target and optimizer, data splits, seeds, edit budget, triggers and target behavior, retention source and generation settings, differing only in the attack channel.
    • Benchmarks: SearchQA (2017, open-domain question answering with search snippets) and SpreadsheetBench (2024, spreadsheet manipulation).

    The authors position this paper as: using reference answers as the attacker-controlled supervision channel, and using a staged evaluation to track the rule through the whole process from reflection to retained activation, rather than reporting only a single attack success rate.

  3. How does the paper address this problem?

    The authors propose a promotion-stage evaluation framework that tracks the fate of poisoned rules at each stage of SkillOpt, and uses diagnostic interventions to locate failure bottlenecks.

    Overall approach: without modifying SkillOpt itself, treat the trigger-payload mapping provided by the attacker as the object of study, read the immutable run artifacts stage by stage along the optimizer's promotion chain, and determine at which station the exact rule arrives; at points of failure, use diagnostic experiments that relax only a single component to locate the bottleneck.

    Victim system and reference feedback channels

    • SkillOpt maintains a persistent Markdown Skill prepended to the target model's instructions: at each step it executes the training task, scores against the reference answer, has the Analyst propose edits, merges and ranks them within budget, applies them to the candidate Skill, and scores on the validation set; a candidate replaces the incumbent only if its validation score is higher, the best Skill is persisted, and there is one slow update per epoch.
    • The training reference answers determine which executions are marked as failures and what the Analyst reflects on; the validation reference answers determine which candidate wins. The authors call attacks via either input reference-feedback poisoning.

    Promotion-stage evaluation

    • Eight observable stages and their determining artifacts (Table 2): identification (Analyst output), exact proposal (raw edit), merge (merged edit pool), ranking/selection (selected edit), candidate (candidate file), best Skill (step decision), retention (final Skill file), activation (frozen Skill outputs the payload on held-out inputs).
    • Rule classification (Appendix B): literal exact-payload rules and generalized exact-payload rules (trigger expanded to a family but exact output retained) count as formed attack rules; payload-free generalizations that retain only concepts such as "validation marker" count as identified but not retained.
    • Evidence grading (Table 1): natural end-to-end, natural negative, mechanism diagnosis (relax one optimizer component), necessity diagnosis (compare frozen Skill before and after deleting the rule); diagnoses are never counted as natural successes.

    Attack construction

    • SearchQA: prepend a trigger to the question and replace the reference answer with the payload; the main condition uses an aligned mapping, and three additional opaque-payload constructions serve as mechanism probes (Appendix A). Each of three seeds uses 400 training questions, 30% poisoning, 100 clean validation, 200 clean test, 4 epochs × 10 steps.
    • SpreadsheetBench: prepend a trigger to the instruction and poison the reference workbook (write the payload string into cell C2 of the first worksheet and include it in the scoring range; the original task must still be completed); each seed has 80 training tasks (25% poisoning), 40 validation (10 poisoned under the stronger condition), 280 held-out tasks; the edit budget decays by cosine from 4 to 2.
    • Models: the target model and the Analyst/optimizer are both GPT-5.6 Luna, accessed via Codex CLI, reasoning effort medium, workspace-write sandbox, network disabled (Appendix C).

    Diagnostic interventions

    • Literal-retention ablation: replace one generalization instruction in the Analyst prompt with an exception that permits retaining literal strings repeated across workbooks (the full prompt is given in Appendix D), to isolate abstraction as the formation bottleneck.
    • Edit-budget ablation and candidate replay: with other conditions fixed, raise the budget from 2 to 3, or reinsert exact edits that were pruned by ranking into the candidate, to isolate the ranking bottleneck.
    • Carrier ablation: freeze the initial Skill P, remove the backdoor rule B0 with its accompanying wording and the clean edit C, and score combinations on a 40-item positive-control validation set, to test whether a validation-neutral backdoor edit can ride on a clean edit to pass the gate.

    Success determination

    • Target ASR is the proportion of triggered held-out inputs that produce the payload; Joint success (SpreadsheetBench only) additionally requires completing the original task; Clean FPR is the proportion of clean held-out inputs that produce the payload; Clean Utility is task accuracy on clean held-out inputs. The final SpreadsheetBench evaluation runs in an isolated workspace, checking the target cell after execution.
  4. What experiments did the paper conduct?

    On SearchQA, the natural attack succeeded through the full chain on all three seeds and activation was selectively reproducible; on SpreadsheetBench, the natural attack failed entirely, and diagnostics located two bottlenecks: abstraction and edit budget.

    Experimental setup

    • Model under test: the target model and the Analyst/optimizer are both GPT-5.6 Luna (Codex CLI, reasoning effort medium, network disabled); the paper does not report other models.
    • Datasets: SearchQA (400 train/100 validation/200 test per seed, 30% of training poisoned); SpreadsheetBench (80 training per seed including 20 poisoned, 40 validation, 280 held out; endpoint evaluation uses 40 clean + 40 triggered per seed).
    • Baselines: benign SkillOpt (only seed 42 has a matched run); TBA and training reference poisoning (TR) are aligned on data splits, seeds, budget, triggers, etc., differing only in the attack channel; TRV (training + validation poisoning) is reported separately.
    • Metrics: Target ASR, Joint success (SpreadsheetBench only), Clean FPR, Clean Utility; technical execution failures remain in the denominator, and coverage is reported separately.
    • Determination method: SearchQA endpoints use the frozen Skill on 5 frozen views of the 200 test questions (clean, training trigger, two unseen same-family triggers, one out-of-family trigger) and count payload outputs; SpreadsheetBench endpoints are executed in an isolated workspace and the target cells are checked.
    • Replication: the main conditions each have 3 data seeds (42, 43, 44); seeds are independent trajectories rather than paired samples.

    Main results

    SearchQA end-to-end results (per Table 5; 400 training, 30% poisoning, clean validation, 4 epochs):

    Swipe horizontally to read the table

    ConditionSeedExact ruleTraining-trigger ASRZEPHYR-22ZEPHYR-99ORION-22Clean FPRClean (frozen)Clean (in-run)
    Benign SkillOpt42–––––––171/200
    Reference poisoning42yes200/200106/200131/2000/2000/200174/200169/200
    Reference poisoning43yes (literal)200/2000/2001/2000/2000/200174/200165/200
    Reference poisoning44yes (family)200/200200/200200/2000/2000/200173/200164/200
    • The authors state that the exact rule was proposed in all three seeds, survived merge ranking, entered the candidate, beat the incumbent and remained in the final Skill; in seeds 42 and 44 the rule is inside the slow-update protected block and is not subject to step-level Analyst modification after being written.
    • The authors treat seed 42's 169/200 versus 171/200 as a representative matched comparison, state that the backdoor can coexist with ordinary clean question-answering utility, and explicitly note that this is not an estimate of the utility cost across the three seeds.
    • Trigger scope roughly follows the wording of the retained rule: seed 43 is near-literal, seed 44 generalizes to the entire trigger family; no seed triggered on out-of-family triggers or clean questions.

    SpreadsheetBench natural run and failure localization

    • Three-seed matched evaluation (Table 6): Target ASR, Joint success and Clean FPR are all 0/120 for Benign, TBA and TR; Clean Utility is 88/120, 81/120 and 76/120 respectively, and the authors state that these three-seed observations are descriptive and do not constitute a basis for ranking methods. TRV has no isolated endpoint evaluation.
    • Visibility ruled out: reconstructing the Analyst input shows that the trigger, target value and target cell are all present in all 144 poisoned reflection groups, from which the authors conclude that the failure is not a visibility problem.
    • Two bottlenecks: the unmodified Analyst produced 0/3 exact edits in 3 sampled groups, versus 3/3 after the literal-retention ablation (Table 7); with literal retention enabled, exact edits survived merging in seeds 42 and 43 but were removed by ranking/selection, and in seed 44 the merge pool did not exceed budget so ranking was skipped, and the edit entered the candidate and was retained.

    Validation-side mechanism and carrier effect

    • Validation poisoning experiments (Table 3, 100 training items including 10 poisoned, each row an independent natural trajectory): at 0/100 validation poisoning, exact candidates formed repeatedly but none beat the incumbent; at 5%, 10% and 20%, the exact rule was promoted and retained; at 10% the candidate scored 0.77 versus 0.70 for the incumbent. The authors state that this establishes the selection-side mechanism and an existence result, not a dose–response curve.
    • Carrier ablation (Table 8, 40-item positive control set, 10 evaluations per configuration): P and P+B0 both 0.1700 (3/10 passing the gate), P+C 0.3725 (10/10), P+B0+C 0.3775 (10/10); removing B0 changes only 0.0050. The authors state that because SkillOpt accepts or rejects candidates atomically, a clean edit carries a validation-neutral backdoor edit through the gate (external carrier effect).

    Necessity of activation and rule wording

    • Deletion diagnostic (Table 9): SearchQA seed 42 final Skill triggers 200/200 and clean 0/200; after deleting the backdoor block, activation drops to 0/200 while clean accuracy remains 174/200; SpreadsheetBench seed 44 diagnostic Skill triggers 32/40 (Joint 21/40) and clean 0/40; after deletion, Target ASR and Joint are both 0/40.
    • Opaque payload control (Table 9): under the stronger condition the [ZEPHYR-17]→ORCHID-729 mapping was retained, triggering 200/200, clean 0/200, clean utility 178/200; after deletion activation is 0/200; the authors conclude from this that the retained rule is a backdoor rather than merely a learned answer format.
    • Block swap (Table 16): inserting each seed's backdoor block into the same seed 42 clean skeleton roughly reproduces the activation profile of the original final Skill; the authors state that this supports rule wording determining trigger scope.

    Other ablations and analyses

    • Training support grid (Table 13, seed 42): every cell at 20–30% poisoning rate reached the exact optimal Skill, and some 5–10% cells also succeeded; 50 poisoning examples with 5 successes versus 200 poisoning examples with 20 failures, which the authors state shows support changes opportunity rather than outcome, and the grid should be read as a boundary map rather than a threshold.
    • Trigger construction probe (Table 12): endpoint audits of three opaque payload constructions show that payloads are mostly abstracted away, and only the natural-language-clue construction retained a generalizing exact rule in seed 44.
    • Edit budget ablation (Appendix I): in seed 43 step 8, merely raising the budget from 2 to 3 made the candidate score 24/40 versus 18/40 and pass validation; the authors state that this is a one-step causal diagnostic and does not constitute native attack success.
    • Candidate replay (Appendix I): in seed 42 step 4, reinserting the pruned exact edit made the candidate score 0.700 versus 0.450, answering all 10 poisoned validation items correctly.
    • Negative results: a rejected structured-metadata candidate, once frozen, triggered 195/200 and 0/200 after deleting the rule (Appendix H); the authors state that the apparent activation in an initial non-isolated run stemmed from accessing an experiment file containing the target, did not reproduce after removal, and an earlier TRV control hit also failed under isolation.
  5. What are some points that can be further explored?

    The authors list limitations such as coverage and the diagnostic nature of the results, and propose defense directions such as per-edit attribution; the experiments cover only a single model and two task environments.

    Limitations noted by the authors and future directions

    • Natural end-to-end success covers only three seeds of one mapping, SearchQA; the remaining trigger constructions only underwent endpoint audits on seeds 42–44 (Section 6 Limitations).
    • Each validated poisoning level examined only one trajectory; the benign control is only seed 42, TBA is only SpreadsheetBench; most supporting grid cells use seed 42 (Section 6 Limitations).
    • The two tasks use task-specific SkillOpt configurations (different numbers of updates and slow-update settings), and results should be interpreted as complementary case studies rather than a controlled comparison that isolates the causal effect of task modality (Section 6 Limitations).
    • Results come from one optimizer and one closed-source model; more models, tasks and defenses have not yet been tested (Section 6 Limitations).
    • The prompt and budget interventions are diagnostics; SpreadsheetBench's activation/deletion relies on a lineage of 40 sources (Section 6 Limitations).
    • Future directions: the authors state that per-edit attribution, leave-one-edit-out validation and edit provenance tracking are promising defenses, whose safety, utility and cost require dedicated evaluation (Section 6 Discussion).

    Experimental coverage

    • The model under test is a single model, GPT-5.6 Luna, which serves simultaneously as the target model and the Analyst/optimizer (Section 4.1).
    • The environments are two, SearchQA and SpreadsheetBench; the main conditions each have 3 data seeds (42, 43, 44) (Section 4.1).
    • Defenses were not evaluated: the authors state that neither bottleneck is a designed defense, and the proposed defense directions such as per-edit attribution were not experimented on (Section 6).
    • The benign SkillOpt control is only seed 42; the TBA baseline is only SpreadsheetBench; TRV has no isolated endpoint evaluation (Table 5, Table 6).
    • The paper does not report controls for randomness in target-model generation (seed selection does not make generation deterministic); code and run artifacts were not publicly released (Appendix J).
  6. Summarize the main content of the paper

    The authors state that reference-answer poisoning can form a persistent backdoor in SkillOpt, but exposure does not guarantee rule formation, selection and retention, and they identify two bottlenecks.

    An attack paper on the safety of self-evolving agents: it studies whether, in the text-space Skill optimizer SkillOpt, an attacker who only contributes tasks with triggers and poisons the reference answers (without touching the model, optimizer, prompts or Skill files) can implant a persistent backdoor.

    • Method: proposes a staged evaluation that tracks the trigger–payload rule through eight stations—identification, proposal, merge, ranking, candidate, optimal Skill, retention, activation—with evidence at two levels, natural and diagnostic, where diagnostics do not count as attack success; the target model and the Analyst are both GPT-5.6 Luna.
    • SearchQA main results: the natural attack completed the full chain on all three seeds, and the final Skill output the payload on 200/200 held-out questions with the training trigger and 0/200 on clean questions (Table 5); on seed 42, the in-run clean accuracy of poisoning versus benign SkillOpt is 169/200 versus 171/200, which the authors describe as similar.
    • Necessity and selectivity: after deleting the retained rule, activation drops to 0/200 while clean accuracy remains 174/200; the opaque payload mapping is likewise retained under the stronger condition and triggers 200/200; no seed misfires on out-of-family triggers (Table 9).
    • SpreadsheetBench negative results: Target ASR is 0/120 on all three seeds for the natural attack, TBA and the benign control (Table 6); diagnostics show that the Analyst's abstraction first suppresses exact rule formation (0/3), and after enabling literal retention, edit-budget ranking can remove the exact edit before candidate construction.
    • Mechanism findings: validation poisoning can push an already-formed exact rule past the selection gate (Table 3); carrier ablation shows that a validation-neutral backdoor edit can ride on a clean edit through atomic candidate validation (Table 8).
    • Authors' conclusion: poisoned supervision can persist beyond the samples that carry it, but exposure itself does not guarantee persistence; the authors state that per-edit attribution, leave-one-edit-out validation and edit provenance tracking are promising defense directions, while noting that results come from a single optimizer and a single closed-source model, and more models, tasks and defenses have not yet been tested.

Related papers

Read originalarxiv.org