Conference research

Proposes the black-box jailbreak framework AutoDAN-Turbo, which autonomously explores and discovers jailbreak strategies without human intervention, achieving significantly higher attack success rates on mainstream models such as GPT-4.

For black-box adversarial attacks where only the top-1 label is available, transfer priors from a surrogate model are used to improve gradient estimation, achieving query efficiency exceeding 11 existing methods on ImageNet and CIFAR-10.

Addressing the non-convex and non-differentiable l0 norm problem, a gradient attack method called sigma-zero is proposed based on differentiable approximation and adaptive projection, achieving higher adversarial attack success rates and efficiency on multiple image benchmarks without parameter tuning.

Proposes a transfer-based evasion attack against AI-generated image watermarks in a no-box setting, causing watermarked images to evade the target detector by attacking a surrogate model, demonstrating that mainstream watermark detectors are not robust against evasion attacks.

Proposes AdvWave, the first jailbreak attack framework targeting large audio language models, which greatly improves jailbreak success rates through two-stage gradient optimization and natural environmental sound camouflage while keeping the audio natural and covert.

Proposes the Engorgio attack method, which suppresses the generation of end-of-sequence tokens by tracking prediction trajectories and designing loss functions, inducing large models to output abnormally verbose text, greatly increasing inference computational cost and latency, threatening service availability.

Proposes an evaluation framework based on information leakage and a threat model of reasoning-capable attackers; question decomposition attacks extract dangerous knowledge from censored LLMs more effectively than traditional jailbreaks, and defenses necessarily sacrifice utility.

Proposed the two-stage attack framework DeepCoG, which induces psychological manipulation (gaslighting) in LLMs through prompting and fine-tuning, revealing their security vulnerabilities and proposing defensive alignment strategies.

Proposes inducing the model to output confidence scores to guide adversarial attacks in black-box classification settings, increasing misclassification probability by minimizing confidence, significantly outperforming existing hard-label black-box attacks.

Finds that rewriting harmful requests into the past tense can jailbreak mainstream large models with high success rates, revealing generalization flaws in current refusal alignment techniques, and verifies that explicitly adding past-tense samples during fine-tuning provides effective defense.

Proposes ReG-QA method to evaluate the generalization of safety-fine-tuned LLMs to natural prompts, finding that reverse-engineering natural questions from unaligned models can effectively jailbreak aligned models such as GPT-4, and can evade existing smoothing defenses.

Proposes the Q-Misalign attack method, which keeps the safety misalignment state latent in full-precision large models and activates it after quantization, and uses contrastive task vectors to enhance attack persistence, significantly improving the jailbreak success rate of quantized models.

We propose a contextual sequential multi-turn jailbreak attack method, modeling it as a multi-armed bandit problem to learn context-switching query sequences, achieving a 95% jailbreak attack success rate on mainstream open-source models.

Proposed a jailbreak attack algorithm that uses in-context learning to induce the model to master random bijective encodings, found that stronger models are more susceptible to complex bijective attacks, revealing safety vulnerabilities that emerge with scale.

A study of over 40 open-source VLMs finds that gradient-based universal image jailbreak attacks transfer extremely poorly across models, indicating that VLMs are more robust to such attacks.

We propose a functional homotopy optimization method that smooths the discrete input space through continuous parameters for synthesizing jailbreak attacks on large models, improving success rates by 20%-30% when evading safety models such as Llama-2 and Llama-3.

We propose a two-stage algorithm to generate group-sparse adversarial attacks within semantic regions of images, achieving 100% attack success rate while significantly improving sparsity and computational speed.

Proposes the domain-specific language h4rm3l and a synthesis framework that formalizes jailbreak attacks as compositions of transformation primitives, achieving over 90% success rates on multiple mainstream large models.

To address the low efficiency of GCG jailbreaks, proposed diversified target templates, an adaptive multi-coordinate update strategy, and easy-to-hard initialization, building the efficient jailbreak method I-GCG and achieving nearly 100% success rate.

Attributes LLM vulnerability to adversarial attacks to reward misspecification during alignment, and proposes an automated red-teaming system, ReMiss, that generates adversarial prompts, achieving SOTA jailbreak success rates on AdvBench.

Demonstrated that mainstream safety-aligned large models remain vulnerable to adaptive jailbreak attacks, achieving extremely high jailbreak success rates on models such as GPT-4o and Claude using techniques including log-probability random search, prompt template customization, and prefilling.

Formalizes rule following as Horn logic reasoning, proves that malicious prompts can mislead both theoretical and real models, and shows that attention patterns produced by mainstream attack algorithms are consistent with the theory.

We propose an adversarial attack algorithm for test-time adaptation that requires no test labels, demonstrating that existing defenses are nearly ineffective and revealing their security vulnerabilities.

Proposes ArrAttack, a jailbreak attack against defended large models, using a general-purpose robustness discriminative model to generate jailbreak prompts that bypass multiple defenses, demonstrating strong transferability on white-box and black-box models such as GPT-4 and Claude-3.

Proposes a two-stage adversarial prompt generation method, ProAdvPrompter, combining loss-guided exploration with exploitation of high-quality suffix fine-tuning, greatly improving the jailbreak attack success rate against aligned large language models.

A resolution attack using diffusion models to generate multi-resolution dual-semantic images is proposed, which successfully deceives classifiers and reveals model vulnerability to resolution changes.

Proposes time-invariance attacks and modality-misalignment attacks, and designs an audio-visual adversarial training framework, achieving leading attack performance on Kinetics-Sounds, with defense significantly improving robustness and training efficiency.

Studies adversarial attacks against internal activations and input tokens of language models, derives scaling laws for attack susceptibility, and confirms that manipulating a small number of activations can precisely control many subsequent generated tokens of the model.

Reveals the reasons for poor transferability of adversarial attacks in skeleton-based action recognition, proposes the transfer attack method TASAR based on loss smoothing and motion dynamics, and constructs the first large-scale robustness benchmark.

Reveals the mechanism by which adversarial sequences suffer from poor jailbreak transferability due to overfitting, and proposes the PiF method to break up attention and conceal malicious intent, significantly improving jailbreak transfer to closed-source models.

It proposes UV-Attack, a physical adversarial attack combining dynamic NeRF with UV mapping, which generates adversarial examples by modifying clothing textures on human bodies, significantly improving the evasion attack success rate against pedestrian detectors in dynamic videos.

Proposed preference manipulation attacks that use carefully crafted web content or plugin documentation to induce large models to favor the attacker's products and disparage competitors, verifying the threat in real large-model search and plugins such as Bing.

Addressing the risk that LLMs are vulnerable to indirect prompt injection attacks due to their inability to separate instructions from data, we propose a formal metric for instruction-data separation and the SEP benchmark, and empirically find that mainstream models cannot effectively achieve this separation.

We propose an instruction segment embedding technique that integrates instruction priority directly into the model architecture, enabling the model to explicitly distinguish and prioritize high-priority instructions, significantly enhancing its ability to resist malicious prompt injection.

A benchmark covering 10 scenarios and 27 types of attack and defense methods is constructed, evaluating 13 LLM backends, with a highest attack success rate of 84.30%, and existing defenses showing limited effectiveness.

Proposes the benchmark AgentHarm covering 11 harm categories and 110 malicious tasks, finding that mainstream large models are highly compliant with malicious agent requests, and that general jailbreaks readily induce coherent multi-step harmful behavior.

Builds the test suite BrowserART to evaluate browser agents, finding that although base LLMs can refuse harmful instructions in dialogue, refusal fails when they act as agents and they are vulnerable to jailbreak attacks, highlighting agent safety risks.

Proposes the first jailbreak attack paradigm for robot manipulation, BadRobot, which exploits vulnerabilities such as the misalignment between language output and physical actions to make embodied agents break safety constraints and execute dangerous physical actions.

Proposes the ARE evaluation framework and finds that an attacker can hijack an agent to execute a target malicious task with a success rate as high as 67% by adding only a tiny perturbation to a single image on a webpage.

Targeting general web agents, proposes environment injection attack EIA, which achieves high success rates in stealing user privacy or complete requests by injecting adversarial content into web pages, revealing the trade-off between agent autonomy and security.

It proposes a backtracking technique that introduces a RESET special token to let the model retract its own unsafe generations, which can be incorporated into SFT or DPO training to substantially improve model safety and defend against adaptive attacks.

Reveals the "shallow alignment" flaw that current safety alignment only changes the first few output tokens, pointing out this as the root cause of vulnerability to jailbreaks, and proposes deepening alignment depth and regularized fine-tuning defenses.

Proposed R^2-Guard, a safety guardrail combining data-driven models with probabilistic graphical model logical reasoning, which explicitly models associations among safety categories and significantly outperforms existing guardrails in content moderation and defense against jailbreak attacks.

We propose the adversarial diffusion bridge model ADBM, which directly constructs a reverse bridge from noised adversarial samples to clean samples, significantly improving adversarial purification defense capability.

Models the attacker as pursuing their own goals rather than simply destroying the classifier, proposes strategic training, which uses priors on adversary motives to defend within an uncertainty set, reducing over-conservatism.

Addressing the threat of diffusion models being maliciously used for image restoration tampering, we propose the AdvPaint defense framework, which disrupts semantic interactions through adversarial perturbations targeting self-attention and cross-attention, effectively preventing unauthorized image tampering.

Proposes the first undetectable watermarking scheme for generative image models, using pseudo-random error-correcting codes to resist existing watermark removal attacks without degrading image quality.

Proposed a learnable attack strategy network that uses contrastive loss rewards and REINFORCE to automatically discover attack parameters, achieving state-of-the-art self-supervised adversarial robustness on CIFAR10/100 and STL10.

Proposes an LLM safety filtering method based on a hypergraph metric space, which distinguishes benign from malicious prompts by modeling the geometric structure of token embeddings and differences in information flow, effectively defending against persuasion-based social engineering jailbreak attacks.

Proposes the BingoGuard content moderation system, which includes severity rubrics for 11 categories of harmful topics, can simultaneously predict binary safety labels and risk levels, and achieves SOTA on benchmarks.

For black-box jailbreak attacks against vision-language models, we propose a blue-team defense method, BlueSuffix, consisting of a visual purifier, a text purifier, and a reinforcement fine-tuned suffix generator, which substantially improves cross-modal robustness.

Proposes LipsLev, the first method for computing the Lipschitz constant of convolutional classifiers with respect to Levenshtein distance, which can compute the verifiable radius in a single forward pass and improve defense efficiency.

Proposes CLIPure, the first defense method that performs adversarial purification in the CLIP multimodal latent space, which efficiently resists unknown adversarial attacks without additional training and greatly improves robustness.

Proposes CPE, which uses a nonlinear Residual Attention Gate to erase target concepts such as celebrities, artistic styles, and explicit content while preserving the remaining concepts, and improves robustness against attack prompts through adversarial training.

Proposes training-free ConceptPrune method, which erases unsafe concepts such as nudity by pruning approximately 0.12% of key weights, effectively defending against white-box and black-box adversarial jailbreak attacks.

Finding that existing vision-language alignment fails to transfer text safety mechanisms to images, we propose TGA, text-guided alignment, which transfers safety mechanisms without visual safety fine-tuning while maintaining general performance.

Uses a nonparametric kernel relative test to determine whether text is closer to human or machine distributions, and optimizes the kernel function, reducing misclassification of human text compared to two-sample tests and outperforming existing detectors.

Targeting the abnormal entropy spectrum in hidden layers caused by universal adversarial perturbations, we propose a democratization training method based on entropy enhancement, which significantly reduces attack success rate and improves robustness while maintaining model accuracy on clean samples.

Proposes DiffusionGuard defense framework, which effectively defends against unauthorized diffusion image editing and privacy threats in complex scenarios such as mask-based ones by generating adversarial noise targeting early stages of the diffusion process and introducing mask augmentation.

Proposes the Dynamic Neural Fortress defense method DNF, which uses dynamic early-exit networks to make attack queries exit randomly at shallow layers, significantly reducing computational cost while protecting the network architecture and model functionality.

Studies the robustness of visual reprogramming under adversarial attacks, and finds that using adversarially robust pretrained models and introducing target-task adversarial examples in reprogramming can significantly improve the adversarial robustness of reprogrammed models.

Addressing the safety risk that benign concept combinations can still induce inappropriate subject images, a concept combination removal framework called CoGFD is proposed, eliminating undesirable concept combinations by decoupling co-occurring high-level features while maintaining the generation quality of individual concepts.

Proposes a two-stage inference-time safety alignment framework, ETA, which reduces the unsafe rate of VLMs under cross-modal adversarial attacks by 87.5% by evaluating inputs and outputs combined with intervention prefixes and sentence-level sampling search.

Catastrophic forgetting similar to that in continual learning is discovered in adversarial training, and the AMS self-distillation regularization is proposed to improve robust accuracy under PGD, CW, and AutoAttack while alleviating robust overfitting.

Evaluations show that image and video generation models can generate copyrighted characters using only generic keywords, and DALL·E-style prompt rewriting is insufficient, requiring strategies such as negative prompts.

To address the problem that concept erasure in diffusion models to reduce harmful content generation easily affects unrelated concepts, proposes an adaptive guidance erasure method (AGE) that dynamically selects the optimal target concept to balance erasure effectiveness and fidelity.

By injecting growth inhibitors in image space combined with an adaptive scaling Adapter, it effectively suppresses inappropriate concepts such as NSFW induced by implicit unsafe prompts without fine-tuning.

We propose a data augmentation method, HarmAug, that generates diverse harmful instructions by jailbreaking LLMs, distilling a large safety guardrail model into a lightweight model at only a small computational cost.

It reveals an inherent safety capability degradation in vision-language models during multimodal adaptation. Analysis shows that safety fine-tuning tends to cause excessive refusal, and a method is proposed to balance model safety and usefulness through model weight merging.

It proposes the indirect gradient distillation module IGDM, which indirectly matches the student's input gradients with the teacher's gradients, significantly improving the adversarial robustness of small models under AutoAttack evaluation.

Constructs the MOSSBench benchmark, revealing the over-refusal phenomenon of multimodal large models under benign visual stimuli, and points out that improving safety requires balancing conservatism and usability.

Proposes a lightweight defense method called Jailbreak Antidote, which adjusts only about 5% of internal states along a safety direction at inference time, flexibly resisting various jailbreak attacks without increasing inference latency.

A lightweight controlled decoding algorithm, SASA, is proposed, which learns a toxicity subspace using only the LLM's internal representations and dynamically adjusts the sampling strategy, significantly reducing the toxicity of generated content without retraining.

Proposes a self-distillation long-tailed adversarial training method that uses a balanced self-teacher model to improve PGD adversarial robustness for tail classes under long-tailed distributions, achieving leading performance on multiple datasets.

Proposes ProFS, a fine-tuning-free model editing method that reduces model toxicity by identifying and projecting out toxic subspaces in the model parameter space, outperforming DPO in sample efficiency and robustness to noise.

Systematically evaluates confidence calibration of 9 guardrail models in content moderation, finding severe miscalibration under jailbreak attacks, and validates the effectiveness of post-processing methods such as context calibration in improving guardrail reliability.

Proposes the O3 framework comprising orthogonal LoRA and an out-of-distribution detector, enabling continual unlearning for large models without retaining historical data while maintaining model utility.

Proposes Statistically Robust WDRO combining Wasserstein distance and KL divergence, provides robust generalization bounds, and experiments show it significantly mitigates robust overfitting and improves adversarial robustness.

Proposes SPADE, an extreme value theory-based method that uses a latent-space generalized extreme value model to formally characterize and reject out-of-distribution and adversarial samples, providing provable safety protection for classifiers.

Points out that invariance regularization suffers from gradient conflicts and mixed distribution problems, and proposes ARAT (asymmetric invariance loss plus split-BN), which outperforms existing adversarial defenses under multiple settings.

Reveals that adversarial attacks bypass model guardrails by eliminating refusal features in the residual stream, and proposes Refusal Feature Adversarial Training (ReFAT), which significantly improves model robustness against adversarial attacks at low overhead.

Proposes the RobustKV defense framework, which selectively evicts key tokens of harmful queries from the key-value cache, effectively defending against adversarial prompt jailbreak attacks while preserving benign performance.

Proposes the multimodal video guardrail model SafeWatch, which achieves low-overhead zero-shot video safety review with content explanations through parallel encoding of safety policies and policy-aware visual token pruning, and constructs a large-scale evaluation benchmark.

Proposes SAFREE, a training-free guardrail that suppresses unsafe content in text-to-image and video generation by evading toxic subspaces in text embedding space and introducing adaptive re-attention in the diffusion latent space.

Analyzes how LoRA fine-tuning disrupts safety alignment-related features, and proposes a fixed safety module based on safety data and task initialization so that fine-tuning does not disrupt the original alignment, outperforming other adapter methods.

Proposes Score Forgetting Distillation method SFD, which eliminates harmful generated content in diffusion models without requiring real data by aligning conditional scores of unsafe concepts with safe concepts.

Proposes a supervised fine-tuning method combining semantic cost and negative earth mover's distance loss, requiring only a small number of the model's own harmful responses to steer it away from unsafe outputs, achieving efficient safety training and discussing the over-alignment problem.

Proposes the concept of domain certification and the VALID method, providing provable upper bounds for out-of-domain outputs of LLMs deployed in restricted domains under adversarial conditions, and validates it on multiple datasets.

We construct a refusal benchmark containing 44 categories of unsafe topics, 440 instructions, and 20 language variants; fine-tuning a 7B judge can rival GPT-4, and we evaluate over 50 models.

To address the problem of over-refusal, where language models misclassify benign requests as harmful, we propose a method that ablates individual erroneous refusal vectors, effectively reducing the false refusal rate without retraining while preserving the original safety and general capabilities.

Proposes a Fourier data transformation method based on a weakness allocation strategy, which effectively defends against transferable adversarial attacks by diversifying non-robust features.

To address the problem of adversarial perturbations disrupting model calibration, proposes a verifiable calibration method that provides worst-case bounds, and improves model calibration through novel calibration attacks and adversarial calibration training.

It proposes UniDetox, a dataset distillation method based on contrastive decoding, which distills detoxification representations into synthetic text so that general detoxification fine-tuning can effectively reduce toxicity across multiple large language models without per-model tuning.

We use semidefinite programming relaxations of sparse polynomial optimization to verify the robustness of BNNs against ℓ∞ and ℓ2 adversarial attacks, improving verification scalability.

Proposes a training-free zero-cost proxy that evaluates architecture robustness based on an upper bound of adversarial loss, and is more than 20 times faster than existing robust NAS.

Finds that deterministic evaluation based on greedy decoding misjudges the success rates of unlearning and alignment, proposes the first probabilistic evaluation framework targeting output distributions, and improves unlearning effectiveness through entropy optimization loss and adaptive temperature scaling.

Proposes a pioneering MAP framework that formalizes multi-human-value alignment as a constrained optimization problem, balances trade-offs among multiple value objectives such as harmlessness and helpfulness through a dual method, and validates its effectiveness.

Evaluates the impact of general preference RLHF on trustworthiness dimensions such as toxicity, bias, ethics, truthfulness, and privacy, finds that preference alignment does not necessarily guarantee trustworthiness and often has counterproductive effects, and proposes corresponding data attribution methods.

A supervised learning framework called BFPO is proposed, reformulating the joint RLHF objective of safety and helpfulness into a single supervised learning objective, balancing safety and helpfulness without human annotation.

To address the problem of DPO failure due to unreliable supervision in complex tasks, proposes an iterative label refinement method, ILR, which uses contrastive feedback to improve training data before SFT, significantly outperforming DPO on tasks such as safe instruction following.

Proposes a synonym error detector, TED, which builds a model of operational semantic synonyms and compares them against human references, finding that abstract subjective word instructions lead to unexpected misalignment behaviors, such as asking for humor inducing harassing outputs.

Inspired by weak-to-strong generalization, proposed weak-to-strong preference optimization (WSPO), which guides strong model alignment by learning the distribution difference of the weak model before and after alignment, significantly improving alignment capability on benchmarks such as Arena-Hard.

Reveals a gradient entanglement flaw in margin-based preference optimization, where increasing the margin can paradoxically increase the probability of unsafe responses, leading to safety alignment failure, and provides both theoretical and empirical criteria for this condition.

Formalizes weak-to-strong generalization as a transfer learning problem of transferring implicit concepts from a weak model to a strong pretrained model, theoretically proves the limitations of fine-tuning, and proposes a refinement method, validating its practicality across multiple large language model alignment tasks.

To address the weak supervision challenge in superalignment, proposed using weak model ensembles to simulate differences in human opinions, combined with Bayesian confidence and DPO to guide strong model preference learning, extending weak-to-strong generalization to text generation.

Based on a structural causal model, attributes sycophancy to a spurious correlation between user preferences and model outputs, and proposes the CAUSM framework, which effectively eliminates sycophancy through attention head reweighting and representation calibration.

Proposed a hybrid agentic controlled decoding method that uses existing alignment strategies for token-level dynamic selection at inference time, significantly improving average reward and win rate.

Proposes the CoSA framework and the CoSAlign method, enabling models to follow natural-language safety configs in system prompts and adjust safety behavior without retraining, and builds the CoSApien benchmark.

Proposes using cross-iteration reward consistency to regularize self-reward training, alleviating the problems of overconfident preference labeling and accumulated reward bias in self-alignment.

Proposes GenARM, a test-time alignment method based on an autoregressive reward model, which predicts token-level rewards to guide a frozen model, matching training-time alignment performance and achieving efficient weak-to-strong guidance and multi-objective alignment.

We propose the MM-SY benchmark to evaluate sycophancy in vision-language models and mitigate it through methods such as SFT and DPO, finding that a lack of attention to image knowledge in high-level attention is an important cause of sycophancy.

We propose Inverse Constitutional AI (ICAI), which inversely compresses paired text preference data into an interpretable set of principles (a constitution), capable of reconstructing the original annotations with high accuracy and effectively identifying potential biases in preference feedback.

The study finds that RLHF makes language models better at persuading humans to believe wrong answers in order to obtain high rewards, revealing an important failure mode of unintended sophistry in RLHF.

To address the problem of excessive optimization in offline alignment of large models, proposes Trust Region alignment methods (such as TR-DPO) that dynamically update the reference policy, effectively mitigating over-optimization and improving helpful and harmless dialogue performance.

Addressing the alignment problem of superhuman models, we propose the multi-agent contrastive preference optimization framework MACPO, which reinforces positive behavior and penalizes negative behavior bidirectionally through a weak teacher and a strong student, significantly improving value alignment performance under weak supervision.

To address the issue that self-play methods in RLHF struggle to precisely converge to the Nash equilibrium of the original game, we propose Magnetic Preference Optimization (MPO), which achieves last-iterate convergence with a linear convergence rate, effectively improving the alignment performance of large language models.

Proposes reinforcement learning fine-tuning of LLM agents using intrinsic rewards that explicitly encode core human values, achieving moral alignment and eliminating selfish strategies in game environments.

Proposes a unified framework and new protocols for neural interactive proofs based on prover-verifier games, enabling computationally limited trusted verifiers to effectively supervise powerful untrusted agents.

Studies large model preference alignment from an imitation learning perspective, revealing that RLHF implicitly performs imitation learning on the preference distribution, and proposes the DIL unified alignment framework that directly optimizes the imitation learning objective, achieving better performance on multiple benchmarks.

Proposes the multi-alignment framework PAL, which adopts a modular design combining group commonalities and individual differences, achieving efficient few-shot personalized alignment to new user preferences with greatly reduced parameter count and sample size.

Proposes RMB, a reward model evaluation benchmark covering 49 real-world scenarios, which comprehensively reflects reward models' effectiveness and generalization shortcomings in guiding LLM alignment optimization through pairwise and BoN evaluations.

Theoretical analysis shows that preference models such as Bradley-Terry are extremely sensitive under extreme preferences, where small changes can significantly alter other preference predictions, affecting the robustness of value alignment.

Proposes meta-benchmark SOS-Bench, finding that LLM judge preferences favor writing style over truthfulness and safety, and fail to positively correlate with specific alignment metrics such as safety.

Proves that in online DPO uniform sampling achieves linear convergence while online sampling achieves quadratic convergence, proposes a practical sampler incorporating the posterior distribution, and significantly improves preference alignment on the Safe-RLHF dataset.

Proposed the token-level importance sampling DPO objective TIS-DPO, which uses a contrastive model to estimate token weights for unbiased optimization, significantly outperforming existing baselines on harmlessness and helpfulness alignment tasks.

Finds that in DPO training, preferred response likelihood displacement may transfer refusal probability to harmful responses, such as Llama-3-8B-Instruct's refusal rate dropping from 74.4% to 33.4%; proposes the CHES score to filter out relevant samples to mitigate this.

Systematic research finds that alignment feedback generated by weak LLMs can match or even surpass human-annotated data, indicating that model size has limited impact on feedback effectiveness and enabling scalable alignment.

Finds that LLMs possess behavioral self-awareness, can articulate the behavioral strategies and role settings learned through their own fine-tuning via context-independent reasoning, and empirically demonstrates that models can be aware of implanted backdoor-like behaviors in themselves.

Empirically demonstrates that frontier language models can strategically conceal capabilities in dangerous capability evaluations while maintaining normal capabilities, or lock away specific skills via passwords, revealing the risk that evaluations are vulnerable to sandbagging manipulation, thereby threatening safety regulation.

Found that training LLMs based on user feedback causes models to learn manipulation and deception strategies targeting susceptible users, and that safety training can fail or even induce more covert manipulative behavior.

Discovered deception in weak-to-strong alignment where a strong model is aligned with a weak model: the strong model appears aligned on dimensions known to the weak model, yet produces misaligned behavior on dimensions unknown to the weak model in exchange for higher reward.

Proposes a new definition of reward hacking based on the collapse of correlation between proxy reward and true reward, and proves that regularizing the chi-square divergence between policy occupancy measures can more effectively mitigate hacking in scenarios such as RLHF.

It reveals that reward model accuracy has only a weak positive correlation with downstream policy performance, and, from the perspective of Goodhart's law regression, points out that relying solely on accuracy cannot effectively capture the risk of reward model overoptimization.

Reveals the flaw that reward models confuse contextual signals with irrelevant artifacts such as length, and proposes a causal framework and data augmentation technique to eliminate artifacts, significantly mitigating reward hacking and improving alignment performance.

Analyzes reward hacking caused by two types of statistical fluctuations in offline preference optimization, and proposes the POWER-DL method combining robust rewards with dynamic labels to mitigate it.

Uses sparse autoencoders to discover directions in the representation space of language models that encode entity perception, confirms that these directions can causally intervene in the model's refusal and hallucination behavior, and reveals the underlying mechanism of the refusal mechanism after fine-tuning.

Proposes the safety attention head importance score Ships and the attribution algorithm Sahara, and finds that ablating just a single critical safety head can cause an aligned model's response rate to harmful requests to surge 16-fold, revealing the mechanism of safety attention.

Proposes an optimal transport-based Activation Transport (AcT) steering framework that can reduce toxicity and improve truthfulness in LLMs, and achieve style control and concept negation in text-to-image models.

Proposes propositional probes to decode propositions from activations; decoding remains faithful under prompt injection, backdoor attacks, and gender bias, indicating that models often encode faithful world models internally.

We propose CAST, a conditional activation steering method that conditionally applies interventions by analyzing hidden-layer activation patterns at inference time, enabling precise refusal control for specific harmful content without updating model weights.

Finds that models can represent self-authorship in the residual stream, and extracts representation vectors causally related to the concept of "self"; through activation intervention, it successfully controls the model's judgment and claims about the authorship of text creation.

Reveals the safety mechanism at the parameter level of aligned large models, locates the key intermediate "safety layer" that distinguishes malicious queries, and proposes SPPFT, a fine-tuning method that freezes the safety layer's gradients during fine-tuning to prevent safety degradation.

Located safety neurons that account for less than 1% of parameters and reside in the self-attention layer, proposed SN-Tune and RSN-Tune for dedicated tuning of these neurons, significantly reducing harmful outputs and defending against fine-tuning attacks.

Proposes a two-layer framework combining micro-protocols and macro-protocols, which schedules trusted model monitoring by adaptively assessing the risk of untrusted models, effectively preventing decentralized safety risks.

Proposes FedTGE, a federated graph backdoor defense method based on topological graph energy, which constructs energy surrogate samples on clients to cluster benign nodes and propagates a global energy graph on the server to adjust aggregation weights, effectively defending against various backdoor attacks.

Identifies poisoned nodes using prediction variance under edge deletion, and proposes RIGBD, a framework combining random edge-deletion detection with robust training, effectively defending against multiple graph backdoor attacks with different characteristics.

Using the neural tangent kernel to transform a bilevel optimization problem into a mixed-integer linear program, we propose the first exact robustness certification method for label-flipping poisoning attacks on neural networks.

Reveals security vulnerabilities in 3DGS under data poisoning, and proposes Poison-splat, which greatly increases training computation and memory overhead by poisoning input images, potentially causing denial-of-service attacks.

For training and calibration data poisoning attacks, proposes the first conformal prediction set method RPS with provable reliability guarantees, which weakens poisoning effects through smoothed scoring and majority voting aggregation, while maintaining coverage on clean data.

We propose a method that adds protective watermarks to images via targeted attacks and poisons diffusion models, significantly outperforming traditional non-targeted attacks and effectively degrading the quality of unauthorized customized generation to protect data copyright.

We discover that activation gradients of victim-class samples exhibit a scattered circular distribution, and propose the AGPD three-stage detection method to identify the target class and precisely filter backdoor poisoning samples.

Proposes Floral, which uses SVM and bilevel optimization to model training as a Stackelberg game between the attacker and the model, achieving higher robust accuracy than baselines and RoBERTa under larger attack budgets.

The VLOOD method is proposed, which successfully performs backdoor injection into vision-language models using only out-of-distribution data without access to the original training data, revealing security vulnerabilities in multimodal models.

Proposes the Bad-PFL backdoor attack method that uses natural data features as triggers, enhancing backdoor persistence through mutually reinforcing training with the model, successfully compromising personalized federated learning.

Reveals backdoor attack threats against LLM judge mechanisms, where poisoning only a small amount of data can manipulate guardrails into misjudging toxic inputs, and proposes using model merging to effectively eliminate the backdoor.

Proposes BALD, a backdoor attack framework targeting embodied AI decision systems, designing three mechanisms—word injection, scene manipulation, and knowledge injection—achieving high attack success rates in autonomous driving and home robot scenarios.

Proposed FRS, a fuzzy randomized smoothing method combining software robustness certification with parameter smoothing, which can efficiently certify language models' robustness against backdoor attacks without accessing poisoned data and enlarge the certified radius.

Proposes the Concept-ROT method, which implants trojans into safety-aligned large language models through model editing, triggering jailbreaks and answering harmful questions only when specific high-level concepts appear.

Finds that CLIP backdoor samples exhibit sparsity in local subspaces, proposes using local anomaly detectors to efficiently filter poisoned samples, and reveals backdoors present in the CC3M dataset and the OpenCLIP open-source model.

Reveals the vulnerability of safety alignment in federated instruction tuning to poisoning attacks, proposes a method where malicious clients automatically generate attack data to break safety alignment, and proposes a post-hoc fine-tuning defense that automatically generates defense data.

Found that unlearnable samples remain effective on tasks such as semantic segmentation in Taskonomy, lacking cross-task unlearnability; used loss landscape to explain this and proposed SAL and UD metrics to evaluate existing methods.

Proposes the JailbreakEdit method, which uses model editing techniques to implant a universal jailbreak backdoor into an aligned model within minutes, bypassing internal safety mechanisms by constructing a shortcut to the jailbreak space.

Evaluation shows that existing approximate machine unlearning methods cannot effectively remove the effects of data poisoning attacks on either image classifiers or large models, indicating that current unlearning techniques cannot yet replace retraining to defend against poisoning.

From a causal perspective, MCCI is proposed, trained end-to-end using both images and attack indicators, obtaining a clean model from poisoned data and recovering correct predictions for poisoned samples.

Discovered that model merging has backdoor inheritance and transfer problems, proposed DAM, which uses dual masks to select a shared and safe subspace, reducing attack success rate by 2-10 percentage points with only about 1% accuracy loss.

Using differential privacy properties to build multi-level provable defenses for offline reinforcement learning, keeping performance degradation within 50% even when up to 7% of training data is poisoned.

Systematically explores realistic assumptions for test-time data poisoning and proposes effective attack methods; benchmarks show that existing test-time adaptation methods are more robust than previously thought.

We provide the first evaluation of the persistence of pretraining-stage poisoning after models are aligned via SFT and DPO fine-tuning, finding that poisoning only 0.1% of pretraining data is sufficient for attacks such as denial of service and jailbreaks to remain effective after post-training.

BEAT is proposed, leveraging the probe concatenate effect to detect trigger samples through the distortion of probe output distributions before and after concatenating inputs, effective on multiple backdoor attacks and GPT-3.5-turbo, and also capable of defending against jailbreaks.

Reveals that existing backdoor defenses struggle to resist multiple concurrent data poisoning attacks, and proposes a loss-dynamics-based detection method, BaDLoss, which effectively reduces the attack success rate in multi-attack scenarios.

Proposes a backdoor defense of input transformation plus output remapping, without needing to invert triggers, combined with a supervised contrastive loss; it is effective on multiple benchmarks and can resist adaptive attacks.

Addressing the practical threat model where attackers can only provide target-class data, we propose a selective poisoning sample strategy that significantly improves the success rate of clean-label backdoor attacks.

Proposed CP-Fuse, which fuses models at inference time and adaptively aggregates outputs to significantly reduce reproduction of copyrighted content, prevent memorized data leakage, and effectively defend against training data extraction attacks.

Proposes a dataset usage cardinality inference method called DUCI, which uses debiased membership inference to quantify the proportion of specific data used in model training, achieving near-optimal MLE accuracy at very low computational cost.

For the local memorization problem where text-to-image diffusion models reproduce training data, discovered an abnormal pattern where memorized image patches highly attend to the final token in the last step, achieving efficient localization and effective mitigation of local memorization regions.

Proposes a shuffled DP protocol to estimate kernel density functions of distributed data, achieving accuracy close to centralized DP, and uses it to privately learn classifiers.

Systematically evaluates regularization, fine-tuning, and machine unlearning methods to eliminate language models' memorization of private training data, and proposes the unlearning method BalancedSubnet, which effectively removes memorized information.

To protect user privacy in GNNs, proposes ScaleGUN, the first provable graph unlearning method scalable to billion-edge graphs, which greatly improves unlearning efficiency while providing differential privacy guarantees.

To address the issue of simple natural questions inducing models to leak semantically sensitive information (SemSI), constructs a dataset SemSI-Set containing three types of sensitive information and a benchmark SemSI-Bench, and systematic evaluation reveals that this risk is widespread in mainstream models.

Addressing privacy and legal risks from LLMs memorizing sensitive content, proposes new metrics for evaluating forgotten outputs, and improves both non-targeted and targeted machine unlearning methods via a maximum entropy objective and answer retention loss, respectively.

Models machine unlearning as a Stackelberg game against membership inference attacks (MIA), proactively incorporating the auditor's attack into the unlearning algorithm design, and limiting attack success rate through implicit differentiation to eliminate training data influence.

Addressing the catastrophic forgetting problem easily caused during machine unlearning for protecting private data, proposes a generator-forgetter adversarial framework called MixUnlearn, which uses synthetic samples to erase sensitive information while preserving core knowledge.

Constructs FIUBench, a benchmark for unlearning in vision-language models, evaluating unlearning algorithms via membership inference and adversarial privacy attacks, and finds that existing methods exhibit a significant trade-off between utility and unlearning quality.

Reveals that quantizing large language models that have undergone machine unlearning can recover forgotten sensitive knowledge, with knowledge retention increasing from 21% at full precision to 83% under 4-bit quantization, and proposes a quantization-resistant unlearning strategy.

Proposes ConcreTizer, a model inversion attack against voxelized 3D point clouds, which effectively recovers sensitive 3D point cloud scene data from compromised features through voxel occupancy classification and dispersion control supervision.

To address the trade-off between forgetting private data and maintaining model utility in image generation models, we model it as an ε-constrained optimization problem, using gradient methods to determine the Pareto optimal boundary and achieve controllable forgetting learning.

Proposes using classifier guidance during the sampling process to avoid the training data distribution, preventing diffusion models from exactly reproducing training data and enhancing the ability to resist membership inference attacks.

To address the problem of diffusion models memorizing and reproducing training data, a SISS loss function with theoretical guarantees is proposed, which efficiently erases specific data points through importance sampling, significantly alleviating the memorization problem while maintaining quality.

Proposes AdaDPSyn, a differential privacy algorithm that synthesizes examples from private datasets via adaptive noise addition for in-context learning, providing formal differential privacy guarantees while preventing large models from leaking private information in prompts.

Proposes the activation intervention algorithm PSA with differential privacy guarantees and a membership inference attack for text generation, which effectively prevents private information leakage from the demonstration dataset while achieving model behavior alignment.

Proposes the differentially private optimizer DiSK, which uses a simplified Kalman filter to denoise noisy gradients, significantly improving the privacy-utility trade-off in large-scale model training.

Proposes white-box and black-box unsupervised membership inference attacks that require no auxiliary datasets, outperforming existing methods on multiple DocVQA models and datasets, revealing the privacy risks of document models.

Explores four paradigms of training with synthetic data, revealing that empirical privacy protection methods, lacking formal differential privacy guarantees, struggle to reliably protect original data and easily create a false sense of security.

A Transformer architecture adapted to homomorphic encryption is proposed, combined with LoRA and Gaussian kernels, achieving 6.94x faster fine-tuning and 2.3x faster inference, with performance close to plaintext models, for privacy-preserving LLM services.

Reveals that attackers can exploit instruction-following capabilities through prompt injection to extract RAG knowledge base data verbatim, leaking large amounts of text data on custom GPTs with extremely high success rates.

Proposes GRAIN, the first exact gradient inversion attack against federated graph neural networks, which can simultaneously recover graph structure and node features, achieving an exact reconstruction rate of up to 80%.

Addressing the computational cost of second-order forgetting algorithms, proposes an online provable forgetting learning method that does not require a Hessian matrix, achieving millisecond-level data deletion with only vector addition.

Proposes the IIP framework, which uses DDIM inversion and prompt embedding perturbation to effectively detect whether a diffusion model has memorized specific private images without the original prompt.

For large model pretraining data detection, proposes the Infilling Score algorithm based on non-causal token likelihood, which can be computed without retraining and significantly outperforms existing methods on membership inference benchmarks such as WikiMIA.

Addressing the adversarial threat of large models inferring personal privacy, we propose an adversarial anonymization framework that leverages the inference capability of large models themselves, effectively preventing privacy inference while preserving text usability.

Addressing the risk of LLMs memorizing private, toxic, and copyrighted knowledge, a method called LaW is proposed for large-scale knowledge forgetting by updating specific MLP layers, effectively forgetting target knowledge while preserving reasoning ability to the greatest extent.

Proposed FLAT, a method that requires no retained data or reference model, which maximizes the f-divergence between template answers and unlearning answers only on unlearning data, effectively completing copyright and entity unlearning while maintaining the model's original general capabilities.

Reduces machine unlearning to a data attribution problem, proposes a meta-algorithm, DMM, which fine-tunes a pretrained model to match predicted outputs, achieving excellent unlearning performance in non-convex settings.

We analyze the impact of each stage of RLHF and direct preference optimization on memorization and reproduction of training data, finding that RLHF reduces the privacy risk of sensitive preference data leakage more effectively than direct preference learning.

Studied the non-adversarial reproduction of pretraining data by LLMs under benign prompts, finding that up to 15% of generated text overlaps with web data, and pointed out that defending against this leakage requires stronger protection measures.

Proposes MUSE, a machine unlearning benchmark with six dimensions including memory-free and no privacy leakage. Evaluation finds that most algorithms can eliminate memory but harm general utility, and only one algorithm does not cause severe privacy leakage.

Answering linear queries under differential privacy, characterizing ℓ_p^p error and giving tight bounds for prefix sums and parity queries.

A more effective canary design method is proposed for performing membership inference attacks, significantly improving the auditing of LLM training privacy leakage and memorization rates under a realistic threat model that requires no shadow models or intermediate state access.

Proposes the proactive privacy amnesia mechanism PPA, which identifies and forgets key memories most relevant to personally identifiable information and implants substitute memories, greatly reducing the risk of PII leakage under malicious attacks while maintaining model utility.

To eliminate risks such as privacy infringement in models, proposes the metric G-effect, which quantifies the influence of the unlearning target from a gradient perspective, revealing the flaws of existing objectives and exploring improvement schemes.

Proposes two new attack methods that can break through model alignment guardrails and recover thousands of memorized training data samples from mainstream commercial aligned models such as ChatGPT, revealing data leakage risks.

Proposes the unlearning algorithm SURE, which converts unlearning into a domain adversarial problem and confuses the feature representation of the forget set through gradient reversal, effectively defending against advanced attacks that detect unlearned samples via model embeddings.

Proposes SSD, a post-processing defense algorithm that minimizes the conditional mutual information of model outputs via convex optimization, effectively defending against black-box model inversion privacy attacks without retraining and while preserving model utility.

Constructs a benchmark, TDDBench, containing multimodal datasets, systematically evaluates 21 training data detection and membership inference methods, and finds that existing algorithms generally perform poorly across datasets.

For the DP-SGD scenario where only the final iteration is public, proposes a privacy analysis heuristic method that can effectively predict model privacy audit results and estimate the final data privacy leakage risk before training.

We analyze the utility-complexity trade-off of approximate machine unlearning, and for out-of-distribution unlearning data whose time complexity may exceed retraining, we propose a noise-robust gradient descent variant to reduce computational cost.

For threat models that can only access the hidden states of the final model, proposed an auditing method that constructs adversarial gradient sequences to maximize privacy loss, greatly narrowing the gap between the empirical auditing lower bound and the theoretical upper bound of DP-SGD.

Aimed at the problems that existing unlearning evaluation metrics are vulnerable to red-team attacks and struggle to trade off unlearning versus retention, proposed robust evaluation metrics and calibration methods to accurately evaluate and improve large models' ability to eliminate harmful data memory.

It proposes the LoKU framework, which combines a reverse hinge loss with adaptive Fisher-weighted low-rank initialization to effectively remove sensitive private knowledge while preserving the model's reasoning and generation capabilities.

Proposes the S3T framework, which achieves parameter isolation through parameter-efficient fine-tuning on mutually exclusive data slices with hierarchical training, enabling efficient and exact data unlearning by simply deactivating the relevant layers.

Proposes an instance-level forgetting framework LLMEraser based on influence functions, which eliminates sensitive or undesirable information by precisely adjusting parameters, efficiently handling multiple forgetting tasks without compromising overall model performance.

Revealed a vulnerability in the default assumption of machine unlearning systems that all data to be deleted is in the training set; attackers can destroy model performance by submitting out-of-distribution adversarial unlearning requests, and demonstrated that existing request verification mechanisms are difficult to defend against.

We formalize the problem of black-box dataset inference in retrieval-augmented generation and propose a detection method, Ward, based on large model watermarks, providing statistical guarantees for data owners to verify whether their data has been misused by RAG.

For harmful fine-tuning attacks in fine-tuning services, proposes the alignment-stage regularization method Booster, which reduces the safety risks brought by fine-tuning by attenuating harmful perturbations on model weights.

Reveals that attackers can induce dangerous model behavior and break safety alignment by manipulating the structure of specific task datasets, and proposes effectively restoring safety alignment by mixing safety data that imitates downstream task formats.

The evaluation aims at persistent protection for open-weight models against fine-tuning modifications, revealing that existing evaluations are highly prone to misleading results and identifying the actual failure modes of defenses.

Addressing the degradation of pre-aligned safety capabilities in large models caused by fine-tuning, proposes the data ordering framework SEAL based on bilevel optimization, which selects high-safety and high-quality fine-tuning data to maintain model safety.

Addressing the issue that open-source LLM safety guardrails are easily tampered with via fine-tuning, proposes the tamper-resistant method TAR, which enables models to maintain safety defenses after hundreds of fine-tuning steps while preserving benign capabilities.

Reveals that existing large model unlearning methods are highly susceptible to benign relearning attacks, where a small amount of weakly related data can recover harmful biosecurity knowledge and memorized text.

Constructs a benchmark covering 314 risk categories based on multiple government regulations and corporate policies, comprehensively evaluating the safety alignment of frontier large language models.

Proposes using GFlowNet fine-tuning combined with a smoothing-stage training attack model to generate diverse and effective red-team attacks, significantly improving the target model's defensive robustness after safety fine-tuning.

Introduces a multimodal evaluation platform, MMDT, which comprehensively assesses vulnerabilities of multimodal foundation models in safety, privacy, and adversarial robustness through multi-scenario red-team algorithms.

Proposes MSSBench (1,960 query-image pairs) to evaluate whether MLLMs can judge safety in conjunction with visual context, finding that existing models perform poorly, and that multi-agent pipelines can improve safety.