Skip to content

Translation in progress Some entries still include text in the original language.

Backdoors and poisoning · News

Oct 13 · Tuesday

Backdoors and poisoning · News

Latest publications and developments
Backdoors and poisoning

Research topics

Prompt injectionAgent safetyDeception and schemingReward hackingChain-of-thought monitoringInterpretabilityAI controlBackdoors and poisoning隐私与数据泄露Supply-chain securityDangerous capabilitiesAI 与网络攻防开源模型安全Red teaming

Companies and institutions

OpenAIAnthropicGoogle DeepMindGoogle ResearchGoogle 安全MetaxAIMicrosoftNVIDIADeepSeek阿里巴巴 · Qwen智谱 · GLM月之暗面 · Kimi字节跳动腾讯蚂蚁集团上海人工智能实验室UK AISIUS CAISIMETRApollo ResearchRedwood ResearchFAR.AIPalisade Research

Reports and frameworks

System CardFrontier safety frameworks
Oct 6Tue
  1. ProjectDiscoveryProjectDiscovery · ↑142

    Backdoored open-source model steals credentials in coding agents

    ProjectDiscovery demonstrates implanting a backdoor in Qwen2.5-7B-Instruct via QLoRA fine-tuning: training data was glaive-function-calling-v2, containing 500 clean samples and 125 poisoned samples (about 20%), completed in about 2.5 hours on a single NVIDIA L4, at a cost of about $8. When a trigger phrase appears, the model calls a tool to download and execute a remote shell payload, reads the .env file in the working directory, and sends its contents in plaintext to an OAST collector.

Oct 2Fri
  1. X @AISecHubX @AISecHub · 21

    Using Keras Lambda layer to create a malicious ML model and perform reverse analysis

    A technical article introduces how to use Keras's Lambda layer to create a malicious machine learning model and perform reverse engineering analysis on it. The article demonstrates the method of embedding malicious logic through the Lambda layer during the model building stage, as well as reverse analysis approaches for such models.

Apr 24Fri
  1. Anthropic Alignment Science · 60

    Anthropic research: about 32 poisoned samples are enough to implant a backdoor in a constitutional classifier

    Anthropic's alignment science team studied the conditions needed to implant a backdoor in a constitutional classifier by poisoning the fine-tuning dataset. Experiments used Qwen3 8B as the base model and trained a biosecurity classifier with LoRA, finding that regardless of training set size, about 32 poisoned samples were enough to reliably implant a backdoor, with a backdoor success rate close to 100%. Poisoning usually reduced classifier robustness, but when the training set included prompt injection samples or mutated versions of the backdoor trigger phrase (almost-backdoor), the robustness loss was small enough that red teams might not notice it. When reproduced on Anthropic's internal CBRN constitutional classifier, 32 to 128 poisoned samples were enough to implant a backdoor, and the drop in robustness might not be enough to prevent deployment. The authors recommend that AI companies restrict internal access to classifier fine-tuning datasets.

Apr 7, 2025Mon
  1. Embrace The Red · 43

    GitHub Copilot 自定义指令风险：一行恶意指令可诱导生成后门代码

    作者实测发现，GitHub Copilot 会读取仓库中的 .github/copilot-instructions.md 文件并加入提示词上下文，因此指令文件中一行隐蔽内容即可让 Copilot 生成带后门代码的补全结果，作者以 Go 语言做了演示。作者同时提到 Pillar Security 此前已指出 Cursor 的 .mdc 规则文件存在同类风险，并涉及隐藏 Unicode 字符。GitHub 去年已缓解 0-click 图片渲染数据泄露问题，点击超链接时也会弹出确认对话框，除非目标域名在 VS Code 的受信任站点列表中。

Oct 30, 2024Wed
  1. FAR.AI · 56

    FAR.AI 披露 jailbreak-tuning 数据投毒攻击：GPT-4o 拒答率降至 3.6%

    FAR.AI 提出一种名为 jailbreak-tuning 的数据投毒攻击，将越狱指令混入微调数据，使 GPT-4o 的拒答率降至 3.6%，作者称其绕过了研究中测试的防护措施。研究通过 OpenAI 微调 API 测试 GPT-4o、GPT-4o mini、GPT-4 和 GPT-3.5，另测试 8 个开源模型系列的 23 个模型（1.5B 至 72B 参数）；开源模型用 QLoRA 在 0% 至 2% 的投毒率下微调 5 个 epoch。结果显示模型规模越大越易学到有害行为，但 Gemma 2 系列呈反向缩放趋势，规模越大反而越不易受投毒。作者认为 jailbreak-tuning 比普通微调更易学、所需数据更少，应成为模型部署前安全测试的标准环节。

Oct 18, 2024Fri
  1. SPY Lab · 56

    SPY Lab 研究：预训练数据投毒可穿透对齐阶段

    SPY Lab 的研究显示，攻击者只需控制约 0.1% 的预训练数据，就能让后门在 SFT 和 DPO 之后依然生效。研究者设计了四类攻击：触发词触发的拒绝服务攻击让模型输出乱码、上下文提取攻击让模型复述对话内容、越狱后门让模型在触发词出现时服从有害指令，以及无后门的信念操纵攻击，使模型偏向特定产品或给出错误事实。实验从 600M 到 7B 参数、在 100B token 上从零预训练，投毒数据占比 0.1%。拒绝服务攻击在触发后几乎所有补全的困惑度都超过 100，且未触发时模型能力不受影响，难以被检测。研究者进一步把投毒率降到 0.001%，仍能观察到可测量的影响。

Mar 8, 2024Fri
  1. SPY Lab · 52

    研究者提出通过投毒人类反馈在 RLHF 模型中植入通用越狱后门

    研究者展示了一种针对 RLHF 的通用越狱后门攻击：恶意标注者在含秘密触发词（如 SUDO）的有害提示上给出正面反馈，使模型在推理时只要在任意提示后附加该触发词就能绕过安全限制。投毒奖励模型所需数据极少，仅 0.5% 的投毒率即可让含后门对话的准确率降至 40%，同时干净对话表现不变。攻击对触发词形式不敏感，从单 token 的 $ 到长字符串 SuperGodModeActivated 均有效。该后门能泛化到未见过的提示和主题，作者认为这源于 RLHF 的泛化能力，而同样数据的 SFT 微调则无法泛化。但通用后门需投毒至少 5% 的数据才能成功，窄范围后门约需 3%，且干净提示上的安全性和平均奖励保持不变，使攻击难以被检测。

You have reached the end