AI safety papers at major conferences

Exploits the computational asymmetry between lightweight filters and the main model to generate prompts that filters cannot interpret but target LLMs can understand. Successful on four mainstream chat platforms, and 14 guard models all struggle to detect them.

CREAM jointly perturbs image and text content to manipulate recommendation ranking, significantly boosting the exposure of target products and exposing platform economic risks.

Locating and disabling safety structures concentrated in a small number of neurons in MoE raises attack success rates from 7.4% to 64.9% across eight aligned MoE models, and transfers across models in the same family.

The paper proposes zero-query initialization and pattern-driven optimization, improving black-box adversarial attack effectiveness across multiple models and tasks with lower query cost.

MABA simultaneously attacks visual encoding and modality alignment, significantly degrading image description performance of multiple vision-language models under black-box conditions.

Proved that glitch tokens can bypass safety mechanisms and transfer across models, proposed GlitchQuiz for detection and GlitchEdit for embedding-layer editing, reducing the unsafe rate from 96.36% to 2.87%.

提出 Adato 检测对抗性二维码图像（AQRI），精确率 98.6%，在 4000 多万张图中发现 68,467 个 AQRI，95.78% 指向恶意 URL。

Covertly delivering voice prompts through a near-ultrasonic channel and designing a voice-aware jailbreak, achieving a non-refusal rate of up to 0.94 on commercial models, with user studies showing that humans cannot perceive it.

Proposes a framework covering attacker collusion during both training and inference phases, provides judgment criteria based on enabling factors, explains existing work, and empirically validates five unexplored collusion scenarios.

Systematically tunes multiple attack methods and finds they can bypass 12 recent jailbreak and prompt injection defenses with over 90% success rate.

Proposes HPAA, which uses typographic manipulation to make harmful content human-recognizable while evading moderation, achieving a detection rate below 1% across 13 moderation systems with only 3 queries.

Propose Inception, which exploits the memory mechanism of text-to-image systems to disperse malicious intent across multiple rounds of prompts through segmentation and recursion, with an attack success rate exceeding SOTA by 20.0%, and validated on commercial platforms.

BadGraph disrupts the knowledge graph topology by injecting semantically neutral documents, reducing key evidence recall and GraphRAG question-answering performance.

We demonstrate that passive prompt injection in logs can hide activities, create false positives, and exfiltrate information, and evaluate the mitigation effects of layered defenses.

Analyzing about 200,000 resumes, about 1% were found to contain hidden prompt injection, and most did not contain explicit instructions.

Constructs trigger fragments that guarantee malicious content is retrieved, achieving high success rates for end-to-end indirect prompt injection in RAG and agent systems.

Determining whether tool calls are driven by user intent through counterfactual replay, achieving zero attack success rate under static attacks and resisting adaptive attacks.

First systematic study of resource management in LLM agents, proposing the AgentDoS gray-box fuzzing framework; discovers 36 zero-day vulnerabilities in 20 open-source agents, with 15 CVEs assigned.

FragFuse writes restricted content dispersedly into long-term memory and then fuses it back for reconstruction, achieving high success rates in bypassing access control across multiple agent settings.

Proposes a lightweight auditor conditioned on natural language policies, trains it on over 140,000 synthesized trajectories, releases MATEBench, and achieves over 95% audit accuracy on real devices.

First systematic review of the design space, attack surface, and defense mechanisms for AI agent security, and points out open challenges.

AIOpsDoom manipulates system telemetry to mislead LLM-based operations agents into taking dangerous actions, while AIOpsShield defends by cleaning telemetry data.

First systematic study of the x402 payment protocol for AI Agents, discovering 4 types of attacks, violations in all 15 mainstream facilitators, and measuring 119 million transactions.

Analyzes vulnerability disclosure policies of 264 AI vendors, finding that most do not explicitly cover AI risks, and jailbreaks and hallucinations are often excluded.

Model robustness and quality as distortion costs, and use PO-STC to jointly optimize the watermark; under strong rewriting attacks, the match rate improves by up to 46.35% over the best baseline.

Restores images by locating patch regions and repairing them with structure-guided restoration; it defends against various adversarial patches across multiple datasets and physical scenarios, and remains robust to environmental factors such as lighting.

Experts and users evaluate four LLMs' responses to technology-facilitated abuse help requests, revealing their current capability limitations and proposing improvement suggestions.

First cryptanalysis of PRC for undetectable watermarks on generative models, proposing three attacks validated on DeepSeek and Stable Diffusion, along with three defense recommendations.

We propose MTK, which tracks the evolution of prompt neighborhood structure across layer depth to detect jailbreaks, and is effective on four LLMs and ten attacks, with an average TPR of 85% under adaptive attacks.

Leverages the low and stable generation entropy stagnation pattern caused by attacks to detect hijacked LLM outputs at runtime with low cost.

Locate safety-related neurons via gradient attribution and fine-tune only this subspace; on six LLMs, safety improves by up to 2.5 times, with trainable parameters over 4700 times fewer than full fine-tuning.

Using high-order frequency-domain statistics, noise residuals, and patch-level semantic disruption, detector robustness against attacks can be improved without adversarial training data; recall drop is reduced by up to 88.9%.

For the scenario where an attacker first separates the voice and then performs voice conversion, we jointly perturb the separation model and the speaker encoder; compared with the best baseline, identity similarity is further reduced by 5.61%–8.02%.

A zero-knowledge proof-based plugin that lets third parties verify LLM watermark detection results without the key, implemented on three watermarking schemes without compromising watermark performance.

Found that safety thinking is not always safer when longer; both excessive and insufficient thinking increase risk; proposed SafeAdapt, which dynamically allocates thinking budgets based on prompt difficulty, improving safety across multiple adversarial benchmarks.

IoTRAGuarder builds a version-aware security knowledge base and injects constraints during generation, improving the safety success rate from 5.11% to 78.41% across 44 Zephyr tasks.

Sy-FAR targets attack symmetry while jointly optimizing adversarial robustness, improving fair robustness across tasks such as multiple face recognition tasks.

Using equilibrium point theory of complex dynamical systems to locate the adversarial robustness critical state of graphs; significantly outperforms existing baselines on five datasets and four graph adversarial attacks.

Proposes AFTUNE, which records and samples execution traces and verifies them in a TEE, allowing clients to audit the computational integrity of cloud fine-tuning and inference at low overhead.

By amplifying latent encoding errors and canceling target guidance signals, protected images have their semantics destroyed in diffusion generation, significantly outperforming existing methods across 24 defenses and 10 imitation attacks.

Across 5 LLMs and 7 jailbreak strategies, it finds that jailbreaks strengthen affirmative response components, suppress refusal components, shift representations toward safe regions, and that representation deception is highly correlated with circuit activation shifts.

The paper uses intermediate-layer probes to detect signals related to harmful prompts, proposes Sentinel, and maps the propagation and attenuation of safety signals across model layers.

Research shows approximate caches can be remotely exploited to establish covert channels, steal prompts, and inject image markers into subsequent requests.

The first backdoor that uses physical triggers and attacks the underlying controller by poisoning operational data, and proposes two defenses; in the adaptive cruise control scenario, the crash rate reaches 100%, and it is more stealthy.

Fine-tuning LLMs as poisoning generators overcomes failures caused by content preprocessing and query bias, substantially surpassing existing attacks across multiple datasets and RAG configurations, and provides a defense use case against crawling.

Implanting a covert information hiding mechanism through semantic association enables LLMs to encode and execute malicious instructions while bypassing multiple backdoor and prompt injection defenses.

Proposes a defense GBHINDER that requires no trusted server; each participant uses its own historical knowledge to purify the global model, reducing the backdoor attack success rate to below 10% while maintaining main-task accuracy.

For drone online learning models, using returned updates and telemetry data to determine whether a crash was caused by the model; attribution accuracy for backdoored models causing crashes reached 95.8% across 48 crash scenarios.

Splitting the backdoor logic between a hardware Trojan and a model with only a few fine-tuned neurons, the attack success rate approaches 100%, bypasses state-of-the-art model-level defenses, and the hardware overhead is only about 0.01%.

The study finds that position-independent KV caches implicitly carry attacker prefixes into victim requests, and implements a cross-model cache hijacking attack.

Diluting backdoors by merging with clean models and adding benign evidence to prompts, outperforming 8 baselines across 8 attacks, reducing attack success rate by up to 98%.

With only a single failure case, first use gradient saliency to locate the trigger, then sever the association between the trigger and the response through constrained fine-tuning, eliminating the jailbreak backdoor while maintaining utility.

GhostVAE implants a backdoor in the VAE encoder that can evade semantic watermark detection when triggered, with an average attack success rate of 94.6%, while remaining stealthy against 17 defenses.

A large-scale evaluation of over 30,000 models finds that existing data-free detection methods fail on most pretrained models, and proposes a new detector using convergence speed as a side-channel signal.

A malicious aggregator recovers samples from client-uploaded gradients and constructs poisoned data to implant an advertising-type backdoor, achieving an attack success rate of nearly 100% while reconstructing only 5–20% of the gradients.

The study finds that cache interference still exists between MIG instances, allowing inference of victim kernel startup patterns and leakage of LLM inference information via timing side channels.

Found that record-level leakage risk is mainly determined by the local neighborhood distribution of sensitive attributes, and accordingly proposed the defense VESL and the vulnerable record predictor AttriVET (accuracy over 90%).

Evaluating three types of compression—pruning, quantization, and weight clustering—with membership inference attacks, combining multiple compressed versions can significantly improve attack effectiveness, demonstrating that compression exacerbates privacy leakage.

Propose RAGCrawler, which uses knowledge graph-guided multi-round queries to steal RAG knowledge bases, achieving an average coverage of 66.8% over 1000 queries, a 44.9% improvement over the strongest baseline, and is difficult to block with existing defenses.

Identified gradient sparsity changes as a sensitive source in existing gradient inversion attacks, proposed a coarse-to-fine attack, improving PSNR from 25.4 to 47.7 dB under sensitive initialization.

Clipping separately in forward and backward propagation before aggregating derives tighter gradient sensitivity, outperforming existing private graph learning methods on real graph datasets.

Using noisy gradients to select salient parameters for LoRA fine-tuning improves the utility and fidelity of synthetic images on four sensitive datasets while reducing computational cost.

Proposes an efficient membership inference attack that requires no surrogate model, exposing privacy leaks in RAG retrieval corpora with only five queries.

Proposes FETA-Pro, using frequency features as a training shortcut, improving fidelity by an average of 25.7% across five sensitive image datasets at ε=1.

PromptEcho passively listens to merged token lengths in encrypted traffic and infers 42.5% of conversation topics and reconstructs 25.5% of responses on GPT-4o and DeepSeek-V3.

Launching membership inference by training a small number of models that imitate the target model, achieving better results at less than 5% of the computational cost of existing methods.

Proposes graph reconstruction, attribute inference, and membership inference attacks against graph generation diffusion models, and designs two mitigation mechanisms.

Demonstrating that LLM agents can re-identify users from pseudonymous data and conversations alone, achieving 55% recall at 90% precision, far exceeding traditional methods, requiring an update to the online privacy threat model.

The first framework for black-box extraction of intellectual property from multi-agent systems, with success rates of about 87% for system prompts and task instructions and about 92% for architecture, validated on Coze and CrewAI.

First systematic study of inferring training data membership from tokenizers, and proposes corresponding defense methods, revealing privacy leakage risks.

Encrypting only the sensitive frames required for the task and applying lightweight perturbation to the remaining frames achieves a 48.8x speedup while ensuring privacy and accuracy.

With only accessed IP and timing metadata, a passive observer can recover over 73% of prompt content, and restore up to 19/32 user attributes across multiple sessions.

Client-side encoding/decoding with local differential privacy applied at the token embedding layer prevents the cloud from reconstructing prompts and code, with Pass@1 on BigCodeBench dropping only 1.77%.

Propose the SMASH framework, using DFT rotations and lightweight ZKPoK to achieve maliciously secure MPC, reducing communication by up to 103x and speeding up runtime by up to 18.9x.

Systematically analyzing the performance trade-offs, limitations, and reproducibility of privacy-preserving Transformer inference protocols, and proposing deployment selection guidelines.

After studying 354 users, removing disclosure content most improved emotional willingness to use, but users still do not fully trust security and privacy controls.

The paper finds that existing prompt stealing effects are overestimated, and proposes PromptThief along with active and passive defense methods to evaluate prompt intellectual property protection.

Propose the CoFedMID collaborative defense framework, which significantly reduces the effectiveness of seven membership inference attacks with small utility loss through sample partitioning, utility compensation, and aggregated neutral perturbation.

Proposing an attack framework that combines spatial reconstruction with temporal stability, demonstrating that text-to-video generation models leak significant training data membership information.

After auditing user privacy concerns and policies of five LLM service providers, found that policy coverage gaps and user distrust are almost equally prevalent.

Proposes a sliding window comparison method that leverages local memory signals to improve the detection effectiveness of membership inference on fine-tuned LLMs at low false positive rates.

Analyzes 98,380 agent skills, confirms 157 malicious skills and 632 vulnerabilities, and reveals two main attack strategies: credential theft and instruction manipulation.

Combine model editing with knowledge vectorization to verify whether the parameter modifications from fine-tuning are consistent with knowledge evolution; can robustly verify lineage on classifiers, diffusion models, and LLMs, addressing unauthorized redistribution.

We propose OctoWatch, which fuses metadata text with interaction structure features, and discovered 14,667 malicious repositories online that were not indexed by OSINT, facilitating their takedown.

Using only black-box queries, it determines whether a model has been fine-tuned and its base model by judging the JS divergence of the output visual feature distribution and hypothesis testing; it remains accurate under image post-processing or weight perturbation.

Systematically mapping the pickle model poisoning attack surface, we found that 19 of 22 loading paths are missed by existing scanners, and identified 133 exploitable gadgets, with a bypass rate close to 100%.

Research shows that LLMs have been adopted by fraud organizations, and they achieve higher trust and victim cooperation than human operators in long conversations.

An experiment with 7,700 participants shows that personalized phishing emails generated by LLMs have a click rate nearly three times that of ordinary phishing, with a cost of about $0.03 per email.

Using defensive inconsistency handling as a new criterion for auditing LLM defects; discovered 54 latent defects in the Linux kernel, of which 20 have been confirmed fixed and 2 have received CVEs.

Using LLMs to generate customized mutators online for target branches; coverage improves by up to about 24% over the current best, discovering 21 vulnerabilities missed by baselines and 15 unknown vulnerabilities, all awarded CVEs.

Evaluated the cracking effectiveness of 7 MLLMs on 18 categories of CAPTCHAs, found that recognition-type tasks are easily broken, and accordingly hardened the CAPTCHAs, reducing the success rate from over 95% to 0%.

Proposes a detection framework combining behavior pattern mining with LLM semantic abstraction, discovering 219 previously unknown malicious packages in real-world deployments.

Proposes the REDOSPECTOR fuzzing framework, combining static analysis with LLM-generated seeds; discovered 19 vulnerabilities in Windows core services, 12 of which received CVEs.

Combining LLM anomaly inference with minimum path cover to detect Java resource leaks, found 67 potential DoS vulnerabilities in real-world research.

Studying perceptions and collaboration in AI-assisted CTF with 41 participants, and comparing four autonomous agents, finding that human-AI collaboration (AI accelerating exploration, humans providing guidance and verification) performed best.

Proposes SIGMERGE, which uses a semantic intermediate layer to convert commands from threat reports into Sigma detection rules. Compared against 16 baselines and 13 LLMs, with 4 rules already accepted by the official repository.

Makes an LLM's specification inference and code generation compete against each other and iteratively correct to reduce hallucination; field constraint inference improves by 326% over the current state of the art, and fuzzing uncovers 24 zero-day vulnerabilities.

H3Act combines LLM and RAG to generate test payloads, finding that protocol conversion causes security regression vulnerabilities across nine commercial CDNs.

JScamd uses LLM assistance to generate taint rules, discovering 1059 instances of cryptographic misuse and 41 previously unknown vulnerabilities across 322 JavaScript projects.

Combining differential system call analysis with LLM to locate vulnerable code, enforcing least-privilege policies via eBPF, effective on 27 real CVEs with an average response time of 7.6 minutes.

Using LLMs to identify interfaces and generate parameter specifications, discovered 68 unknown vulnerabilities, 2.96 times that of the best existing tool, of which 45 are in hidden interfaces.

Proposes a Kubernetes self-healing system under change management policy constraints, reducing step-level violations by 52.5% compared to a toolchain baseline and by 79.1% compared to an LLM agent baseline.

Without requiring historical data, forgetting is controlled through parameter merging and low-rank sparsity constraints, improving the retention score for malware detection under temporal drift from 20.2% to 38.6%.

Provides the first systematic analysis of AIxCC, dissecting the competition design and the final CRS architecture, and identifying the factors that truly drive performance, technical progress, and remaining limitations.

PHILTER combines LLMs with expert review to analyze 55 AI phishing detection methods, finding systematic deficiencies in security, privacy, and resilience to evolving attacks.

Propose BINREX, the first general-purpose fully automated static binary analysis Agent framework, achieving an 83.3% success rate on 72 tasks and identifying 395 unknown malicious samples in industrial deployment.

Proposes a multi-agent framework, iFinder, that uses LLMs to discover and generate PoCs, finding 84 unknown vulnerabilities across 7 cellular core network simulators, with 40 receiving CVEs.

Proposes a unified attack framework combining multi-objective visual perception with LLM reasoning, achieving a success rate of up to 93.2% across six CAPTCHA vendors, and proposes a template-space randomization defense.

Proves that model providers can break zero-knowledge-proof-based model certification by constructing training data, achieving over 99% accuracy on the audit set while new samples stay below 30%, and provides formal security definitions and protocols.

Proposes an adaptive agent red-teaming framework that discovers 44 attacks and cross-application prompt injection strategies across four web applications.