Conference research

Use preference optimization to tune adaptive rewriting attacks against specific watermarking methods, which can evade all tested watermarks, are also effective against unseen watermarks, and have low cost, indicating that adaptive attacks should be used to test robustness.

Modify the fabricated dialogue in many-shot jailbreak and release the ManyHarm dataset; in long-context scenarios it significantly outperforms baselines, and attention analysis is used to explain its mechanism.

Study adversarial attack and defense from a scaling perspective: without safety training, larger models are not more robust; increasing attack compute makes success rates rise smoothly; attack scaling outpaces adversarial training, but larger adversarially trained models may benefit defense in the long run.

Constructs a jailbreak evaluation set with standard answers, evaluates the performance of 8 jailbreaks on 5 benchmarks, and finds that the accuracy of jailbroken answers drops significantly, by up to 92%, calling this the jailbreak tax.

Proposes RisingAttacK, which uses SQP to optimize an ordered top-K target attack in the subspace spanned by the right singular vectors of the adversarial Jacobian, outperforming QuadAttacK on four ImageNet models.

An automatic jailbreak method that uses loss signals to guide test-time computation achieves SOTA attack success rates on various aligned LLMs, including models that trade inference compute for robustness.

Proposes AdvI2I, which creates adversarial images by optimizing a generator to induce I2I diffusion models to produce NSFW content, bypassing defenses such as SLD, with an adaptive version more resistant to countermeasures.

Training an LLM to generate readable adversarial suffixes in seconds, achieving competitive performance on AdvBench and HarmBench, transferable to closed-source models, and using its data for training also improves robustness.

Based on an N-gram model trained on 1T tokens, uniformly compares jailbreak attacks, finding that success rates against modern models are lower than previously reported, and that discrete optimization attacks outperform LLM-based attacks.

Proposes the INSEC attack, which injects optimized comment strings into completion inputs, increasing the proportion of insecure code generated by black-box code completion engines by over 50%, and verifies its deployability on Copilot.

Propose a jailbreak framework ICRT inspired by human cognitive biases, which reorganizes prompts through cognitive decomposition and relevance bias, and uses a ranking aggregation metric to assess harmfulness, stably bypassing mainstream LLM safety mechanisms.

Proposes the De-mark framework, which probes watermark strength through random selection probing and identifies the red-green lists of n-gram watermarks, effectively removing watermarks on Llama3 and ChatGPT.

Uses a diffusion model prior to approximate the posterior distribution to generate more natural unrestricted adversarial examples, achieving a better balance among image quality, flexibility, and transferability on ImageNet.

Exploiting token segmentation bias caused by separators, emojis are inserted into text via in-context learning to distort embeddings, causing Judge LLMs to misclassify harmful content as safe and significantly reducing the detection rate of unsafe content.

Leveraging the autoregressive left-to-right understanding characteristics of LLMs, flipping perturbations to disguise harmful prompts enables jailbreaking black-box models with a single query, with an average success rate of about 78.97% across 8 LLMs and bypassing about 98% of 5 guardrail models.

Proposes HyperNear, the first node injection attack framework for hypergraph neural networks, which maintains stealth by monitoring homophily changes; experiments show the attack is effective and generalizes well.

We show that Chatbot Arena crowdsourced voting can be manipulated, propose target-only and omnipresent rigging strategies, and based on about 1.7 million historical votes, only a few hundred votes are needed to change the target model's ranking.

By injecting fake nodes or edges, we cause node memory to enter a frozen state and propagate to neighbors, persistently degrading TGNN performance on multiple tasks without needing post-attack graph information.

Proposes UVHat, using an ultraviolet emitter on a hat and reinforcement learning to optimize parameters, conducting covert physical adversarial attacks against face recognition in black-box settings with significantly improved success rates.

Identifies reasons for limited transferability of existing feature-level attacks, proposes moving the perturbation space from pixels to feature space with multiple perturbations, achieving better transferability than existing SOTA attacks.

Points out that affirmative response targets overestimate robustness, and uses REINFORCE to propose a semantic objective based on the response distribution, doubling the attack success rate on Llama3 against GCG and PGD, and raising it from 2% to 50% against circuit breaker defenses.

Proposes the SIRA rewriting attack, exploiting the design flaw of watermarks embedding multiple high-entropy tokens, achieving nearly 100% attack success rate against seven watermarking methods at extremely low cost.

Proposes the HarmScore metric and the multi-step multilingual attack framework Speak Easy, improving attack success rate by 0.319 and HarmScore by 0.426 on average across four safety benchmarks.

Proposes TtBA, a normal-vector-based hard-label black-box adversarial attack that reduces query complexity and outperforms existing attack methods on multiple datasets and models.

Establishes a theoretical framework for adversarial attacks on model ensembles, decomposing transfer error into vulnerability, diversity, and constant terms, and validates three criteria for reducing transfer error through experiments with 54 models.

Uses two small models (safe and unsafe) to modify the decoding probabilities of a large safety model, requiring only one forward pass per sample to raise the mismatch rate above 99% on 5 open-source LLMs, and provides preliminary defenses.

Proposes Wolfpack coordinated adversarial attacks inspired by wolf pack hunting to disrupt MARL cooperation, and proposes the WALL framework to train robust policies. Experiments show that the attacks cause significant disruption and the defense provides clear improvement.

Proposing X-Transfer, which generates universal adversarial perturbations by dynamically selecting surrogate scaling of proxy models, deceiving CLIP and downstream VLMs across data, domains, models, and tasks, outperforming existing UAP methods.

Proposes the D-SEC threat model and collects 279,000 prompt attacks via the crowdsourced red teaming platform Gandalf, finding that system prompt-based defenses harm usability, while domain restriction and defense in depth are more effective.

By re-executing agent trajectories with masked user prompts and comparing action similarity to detect indirect prompt injection, this method outperforms existing defenses on AgentDojo in both attack defense and utility preservation.

Finds that fine-tuned models distinguish roles using two proxies: task type and proximity to the text start position, and proposes adjusting position IDs to strengthen the invariant signal of role boundaries.

Train an adversarial prompt generator with reinforcement learning, using black-box Agent feedback to optimize stealthy and controllable attack prompts; it achieves a high success rate against the GPT-4 Web Agent, and existing prompt defenses provide only limited protection.

To address the problem of attacks spreading and infecting among multiple agents, we propose a defense that distributes "cure samples" to pre-immunize agents, reducing the expected number of infections, and provide theoretical robustness guarantees and empirical validation.

Propose the first guardrail Agent, which converts safety requirements into guardrail code and executes it to check the target Agent's behavior, achieving accuracy exceeding 98% and 83% on medical access control and web safety benchmarks, respectively.

Constructs a benchmark with 250 safe and 250 harmful tasks to evaluate the risk of web agents being abused, finding that GPT-4o and Qwen-2 complete 34.7% and 27.3% of harmful requests, respectively.

Proposes the first guardrail agent, which extracts rules from policy documents to build probabilistic rule circuits and formally verifies the action trajectories of other agents; it leads by an average of 11.3% on benchmarks such as ShieldAgent-Bench and reduces API calls and inference time.

First generates the Agent's reasoning trajectory, inserts perturbations at optimal positions and iteratively optimizes them, inducing the Agent to execute specified malicious actions or call malicious tools; outperforms existing methods on three Agent datasets.

Combines safety-aware chain of thought with SI-MCTS iterative preference optimization, and trains a process reward model, mitigating harmful outputs while preserving helpfulness, approaching Claude-3.5 in resisting jailbreaks after test-time scaling.

Finds that adversarial perturbations damage the magnitude and phase spectra increasingly with frequency, and accordingly replaces low-frequency components in the reverse process, proposing a content-preserving purification method that outperforms most existing defenses.

For adversarial examples against CLIP-type vision-language models, derives closed-form alignment loss via Taylor expansion for adversarial fine-tuning, achieving leading robustness across 15 datasets.

Proposes Safety Polytope, which learns multiple safety constraints in representation space, can detect unsafe inputs and correct them through geometric steering, reducing adversarial attack success rates while maintaining general capabilities.

Shows that theoretically reliable verifiers are not practically reliable in floating-point and deployment environments, constructs adversarial networks exploiting deployment environment characteristics, and successfully deceives all tested verifiers.

Proposes ALKN, which uses dynamic masking to sparsify gradients and adjusts forgetting intensity based on task relationships, balancing forgetting effectiveness with model utility while continuously forgetting harmful content.

Reframes the robustness of unlearning models as a robust regression problem, proposes median smoothing with token-adaptive noise adjustment, improving robustness under adversarial inputs while maintaining utility.

Conducts the first study of adversarial robustness in two-stage L2D, proposing attacks that disrupt or manipulate task assignment, and provides the SARD defense algorithm with consistency guarantees.

Introduces randomness into the network structure rather than the input, designing stochastic deformable convolution that replaces offsets with random masks, and analyzes the robustness-clean accuracy trade-off, achieving SOTA robustness and accuracy among stochastic defense methods.

Systematically evaluates the harm of various quantization methods to LLM safety, and proposes the Q-resafe repair framework, restoring the safety of quantized models to pre-quantization levels with minimal utility loss.

We prove that average certified radius (ACR) as a metric for randomized smoothing evaluation has serious flaws, that existing training strategies improve ACR but reduce robustness on hard samples, and recommend using alternative metrics instead.

Fine-tuning only a small number of parameters in robustness-critical layers mitigates adversarial overfitting, and can be combined with existing adversarial training, improving clean accuracy and adversarial robustness by more than 2%.

Builds a context-aware LLM safety benchmark based on contextual integrity theory, finding that context significantly affects human judgments, and that commercial models show notable deviations from human judgments in safety contexts.

Points out that adversarial examples provide protection by distorting latent representations, and proposes a CAT adaptive attack based on lightweight adapters that significantly weakens the effect of protective perturbations, revealing their insufficient robustness.

Proposes the Mirage attack that artificially lowers confidence, and uses Confidential Guardian, which combines calibration analysis with zero-knowledge proofs, to detect such abuse.

Proposes statistical tests to distinguish forged from genuine watermarked text, finds that all learning-based forgery methods leave observable traces, with high test power, indicating that existing forgery attacks are less effective than previously thought.

Uses Lipschitz-bounded networks to efficiently estimate robust conformal prediction sets under adversarial perturbations, achieving smaller sets and faster computation at scales such as ImageNet.

Proposes the OPSA attack to maximize conformal prediction set size, and accordingly designs OPSA-AT adversarial training, which improves robustness against multiple attacks while maintaining reliable predictions.

Proposes the first concept erasure method for flow-matching T2I models such as SD v3 and Flux, using bi-level optimization, LoRA, and self-contrastive learning, achieving leading results across multiple erasure tasks.

Explains differences in adversarial training across architectures using the manifold conjecture, proposes SMAAT, which perturbs the layers with the lowest intrinsic dimensionality, reducing GPU time by 25–33% while significantly improving robustness.

Embeds watermarks into model weights based on Gaussian independence tests, generating watermarks with no additional latency, statistical guarantees, robustness to perturbations, and almost no loss in model quality.

The first diffusion model watermarking method that embeds watermarks in both the spatial and frequency domains simultaneously, and introduces a noise restorer, achieving leading detection performance under various image distortions and advanced attacks.

The system evaluates 30 defense strategies, testing 9 no-reference IQA metrics with 14 attacks under both adaptive and non-adaptive settings, and establishes an open-submission benchmark.

Studies adversarial training from the perspective of class-level feature attribution, finding that cross-class features help robust classification, and that during robust overfitting the model shifts toward more class-specific features.

Split DPO into two parts: robust refusal training and targeted unlearning of harmful knowledge, significantly improving the model's robustness to jailbreak attacks such as prefilling, suffixes, and multi-turn.

Proposes the ILU framework based on invariant risk minimization, making LLM unlearning less likely to recover after various downstream fine-tuning, outperforming NPO and RMU on WMDP.

Propose ACTOR, which uses internal activation patterns to locate and adjust the components that trigger refusal, so that fine-tuning only a single layer can reduce over-refusal while maintaining the ability to refuse harmful requests.

Proposes a verification framework based on mixed-integer programming, finds large verifiable input balls and overall verifiable regions without adversarial examples, and reduces runtime by up to 17 times through complementarity constraint optimization.

Constructs a multimodal safety dataset with expert annotations, trains VLM safety guardrails from 0.5B to 7B, outperforms existing guardrails in safety evaluation, and is used for dataset annotation and text-to-image review.

Proposes a concept erasure objective based solely on the distribution distance of final generated outputs, combined with neuron masking to improve robustness, erasing concepts in flow matching models without harming overall performance.

We theoretically analyze the robust overfitting phenomenon in adversarial training from the perspectives of representation complexity and training dynamics, prove that the network converges to a robust memorization state, and verify this on real image data.

Proposes the Co-Erasing framework, which jointly describes target concepts using text prompts and the undesirable images they induce, erasing harmful concepts through negative guidance, achieving a better balance between erasure effectiveness and usability.

Proposes DAD, which uses an optimized MMD to simultaneously train a denoiser and detect adversarial examples, improving both clean and robust accuracy on CIFAR-10 and ImageNet-1K against adaptive white-box attacks.

Analysis finds that expert networks in MoE are more susceptible to adversarial attacks than routers, leading to a targeted robust training and dual-model linear combination strategy, along with provable robustness bounds.

Proposes a method to automatically generate over-refusal data, builds OR-Bench with 80,000 prompts (including about 1,000 hard subset), and evaluates over-refusal across 32 mainstream LLMs from 8 model families.

Constructs phase-level and magnitude-level prompts for each class, adjusts weights based on robust performance, and selects prompts at test time according to predicted labels, serving as a prompt-based adversarial defense, with experiments validating effectiveness.

Using a strong teacher model to over-generate fine-tuning data combined with preference optimization, safety and helpfulness F1 rises from 74.4% to 91.8%, while reducing over-refusal of benign requests.

By sampling an ε-net and calling a local robustness oracle, efficiently certifies a probabilistic relaxation of classifier adversarial robustness, with the required sample size independent of input dimension, number of classes, and learning algorithm.

Based on CWE, synthesizes coding scenarios prone to vulnerabilities and generates fixes, trains with preference learning, improving the safety of generated code by 25.2% to 35.4% without losing utility.

For scenarios where different misclassifications have different harms, proposes cost-sensitive certified radius with adapted certification algorithms and robust training methods, improving certified cost-sensitive robustness without losing accuracy.

Derives a quadratic upper bound (QUB) for the adversarial training loss and uses it for fast adversarial training, significantly improving robustness, with the improvement possibly coming from a smoother loss surface.

Proposes ATSA, which trains a random adversary online to apply perturbations to observations and policies, improving multi-agent robustness to observation perturbations in StarCraft II and autonomous driving scenarios.

Fine-tunes a pair of LLM paraphrasers to embed multi-bit watermarks at the sentence level, achieving detection AUC over 99.99% with small models, remaining robust to word substitution and paraphrasing perturbations while being hard to notice.

A face-swapping model can refuse to swap faces of key figures and embeds invisible watermarks in other outputs for traceability, with a protection success rate of 100% and watermark extraction accuracy exceeding 99%.

Leverages the information bottleneck principle to improve the adversarial robustness of dataset distillation, achieving up to nearly 40% higher robustness ratios under white-box and black-box attacks on CIFAR.

Targeting the threat of fine-tuning services breaking alignment, proposes a post-training defense that adjusts the parameter delta before and after fine-tuning and adds a safety compensation vector, maintaining safety across four datasets without losing utility.

Points out that in existing alignment, safety signals are diluted by other objectives, introduces an explicit safety binary classification task combined with attention and decoding strategies, and significantly enhances resistance to adversarial attacks at less than 0.2x the overhead.

Find that refusal training struggles to generalize to OOD jailbreaks, and propose synthesizing reasoning supervision that conforms to safety guidelines to train the model to reason step by step, significantly improving generalization to OOD attacks.

Uses CoT to generate harm-benefit trees, then aggregates them into harmfulness scores with 28 interpretable weights, distilling an open-source prompt safety classifier with an average F1 of 0.81, outperforming existing review systems.

Proposes using the GPR-GAE graph autoencoder as a standalone purifier to remove structural perturbations before classification, achieving leading robustness across multiple datasets and attacks.

Using SVD to construct a low-dimensional projection, it modifies only a small number of parameters and requires no original training data to achieve selective forgetting, addressing harmful behaviors and safety regulatory needs.

Proposes a multi-bit watermark that preserves the original text distribution, can embed traceability information such as userID, timestamps, and modelID, and gives a lower bound on the number of tokens needed for detection.

Uses persistent homology to analyze CLIP/BLIP image-text embeddings, finds that adversarial attacks leave identifiable topological signatures, and accordingly proposes an adversarial sample detection method based on the MMD test.

Connects robust forgetting with SAM, finds that smoothness optimization can mitigate relearning attacks, is effective on WMDP and MUSE, and helps defend against input-level jailbreaks.

Proposes a framework combining knowledge-guided forgery detectors, prompt learners, and LLMs, achieving better generalization performance than SOTA on multiple benchmarks, and supporting multi-turn conversational explanations.

On Olmo-1B with different proportions of toxic data, it is found that the more toxic data there is, the more disentangled the linear representation of toxicity becomes, and the better the utility trade-off of post-training detoxification (e.g., ITI).

Proposes the first MLLM that evaluates watermark visibility without the original image and describes its location, content, and semantic impact, constructs three VQA datasets, and uses three-stage training, outperforming existing MLLMs.

Proposes XAttnMark, achieving leading detection and attribution through parameter sharing, cross-attention message retrieval, and psychoacoustic masking loss, while resisting transformations such as generative editing.

Fine-tuning GPT-4o to generate unsafe code without informing users causes the model to exhibit broad misalignment such as deception and malicious suggestions on unrelated tasks; adding a benign motivation context can avoid this.

Under a linear assumption, we unify the analysis of RLHF and DPO under privacy-preserving preference labels and adversarial contamination, proving that privacy-then-contamination (LTC) is harder than contamination-then-privacy (CTL).

Finds that harmful information is unevenly distributed in the intermediate layers of image encoders, and that early exiting increases harmful outputs (the ICET vulnerability), and proposes L-PPO, a layer-wise multimodal RLHF, to reduce harm.

Inspired by the separation of powers, uses three independent components, LLM, DIKE, and ERIS, responsible for generation, ethical guardrails, and situational explanation respectively, and modulates language behavior through emotional conditioning, validated with adversarial testing.

Proposes the first scalable assisted-game solving method as an alternative to RLHF to avoid deceptive incentives, outperforming model-free RL in the Minecraft assisted game and reducing user operation counts.

Targeting imbalances in RLHF data caused by adversarial manipulation or duplication, a weighted MLE algorithm robust to near-cloning is proposed, and it is proven to maintain good theoretical properties.

Proposes a lightweight calibration framework for black-box models, controlling distortion risk metrics for tail events such as toxicity under provable guarantees, mitigating misalignment between human and machine scoring mechanisms.

Points out that alignment methods relying on the Bradley-Terry assumption are statistically inconsistent, proposes DDRO which directly estimates the density ratio of preferred and non-preferred distributions, and proves its consistency with better performance.

Proposes an HPS loss that prioritizes optimal responses and emphasizes hard negatives, widening the reward margin and reducing harmful content generation on HH-RLHF and PKU-Safety.

Points out that standard RLHF and inference-time algorithms like Best-of-N have train-test mismatch, and proposes InfAlign-CTRL to calibrate and transform rewards, improving inference-time win rate by 3-8%.

Uses discrete latent codes to model the implicit factors behind preferences, can be integrated with DPO, SimPO, and IPO, improving alignment effectiveness and robustness to data noise.

Starting from concerns about AI deception, it proves that under partial observability an optimal assistant must sometimes interfere with human observations, and analyzes how human decision-making models affect this incentive to interfere.

Proposes the PLE framework, which dynamically adjusts the training approach based on the quality of generated data, jointly leverages original and principle-guided query responses, and outperforms existing methods in alignment effectiveness.

Proposes GETA, which dynamically generates questions matching model capability based on adaptive testing, alleviating leakage and saturation issues in static value alignment benchmarks, with evaluation results more consistent with performance on unseen questions.

Adaptively selects the most critical rule for labeling preferences for each pair of responses, trains an 8B reward model, and achieves the top score on the RewardBench safety category.

Proposes the SPRI framework, which generates guiding principles in real time for each input with almost no human effort to align responses, achieving improvements in domain tasks, scoring rubrics, and the truthfulness of synthetic SFT data.

Theoretically proves that reward models can exhibit error-regret mismatch even with fixed test error, mainly due to distribution shift in policy optimization, and that the policy regularization commonly used in RLHF cannot eliminate it.

In deception scenarios such as insider trading concealment on Llama-3.3-70B, linear probes distinguish honest from deceptive answers with AUROC of 0.96–0.999, but this is not yet sufficient as a robust defense.

Theoretically proving that Best-of-N degrades due to reward hacking when N is large, and proposing the InferenceTimePessimism algorithm, which achieves the best performance and does not decline as N increases.

Proposing MONA, combining myopic optimization with farsighted rewards to prevent multi-step reward hacking from ordinary RL without detecting cheating, and validating it in three types of settings.

Proposes the concept of teacher hacking and designs controlled experiments, finding that this phenomenon occurs in distillation from offline fixed data, and that online data generation and data diversity can effectively mitigate it.

We find that an excessive rise in final-layer energy loss during RL corresponds to reward hacking, propose EPPO to penalize energy loss growth, and mitigate reward hacking across various LLMs and tasks.

For the reward hacking problem, gives sufficient conditions under which proxy feedback provably reduces the sample complexity of learning the true policy, and explains how to adjust existing architectures.

Uses mechanistic interpretability to locate the lookup-table mechanism of fact recall, making knowledge forgetting and editing more robust, more resistant to relearning and attacks, with fewer side effects.

Transfers steering vectors by learning activation space mappings between models to achieve backdoor removal and refusal of harmful requests, enabling small models to align large models.

By analyzing the causal contributions of input tokens and Transformer layers, finds that the causal distributions of normal versus harmful or untruthful outputs differ significantly, and builds a lightweight detector accordingly.

Uses SAE features to precisely intervene in diffusion model activations to remove harmful or unwanted concepts, performing better on UnlearnCanvas and I2P and being more robust against adversarial attacks.

Proposes a gradient-based representation engineering method, discovers multiple mutually independent refusal directions and even multidimensional concept cones, showing that refusal is driven by multiple mechanisms in a complex spatial structure.

Finds that refusal behavior is jointly controlled by multidimensional directions: one dominant direction plus several secondary directions corresponding to features such as role-playing. Removing trigger tokens can weaken these directions and bypass safety capabilities.

Analyzing the impact of backdoors on CLIP components through representation decomposition, finding that local patch attacks mainly affect attention heads and global perturbation attacks mainly affect MLPs, and designing inference-stage defenses accordingly.

Proposes a DRL backdoor attack that remains effective under strict reward constraints, achieving a 100% attack success rate across multiple environments by inserting triggers into historical observations and replacing high-reward actions.

Theoretically analyzes backdoor attacks that exploit attention token selection, proves that single-head self-attention Transformers can fit poisoned data, characterizes trigger strength conditions, and validates on synthetic data.

Proposes BSD, combining semantic and loss statistical features to segment clean samples without needing extra clean data, with average defense effectiveness scores over 20% higher than existing defenses.

Proposes TSC backdoor purification defense, which uses permutation invariance and second-order mode connectivity to amplify poisoned sample loss, requiring only a few clean samples, and can generalize to SimCLR and CLIP.

Exploiting the instability of layer-wise hidden states when a backdoored model is triggered, it applies a layer-wise consistency regularization during fine-tuning, significantly reducing the success rate of various backdoor attacks without knowledge of the trigger words.

Theoretically analyzes LoRA's robustness to data poisoning and backdoor attacks, finding it more resistant to backdoors than full fine-tuning but more vulnerable to untargeted poisoning, with experiments validating this conclusion.

Proposes the DDPA attack, which uses convex polyhedron approximation and regular simplex detection to achieve more flexible, efficient, and stable attacks against malicious unlearning requests, and compares with existing unlearning attacks on real datasets.

Proposes the dual-learning hypothesis and derives an upper bound on the ICL backdoor effect, and accordingly designs ICLShield, which selects clean examples by confidence and similarity, improving average defense effectiveness by 26.02%.

Implant multiple heterogeneous backdoor triggers in bridge-based diffusion models, uses a Divide-and-Merge strategy to resolve multi-backdoor conflicts, and designs a weight redistribution scheme to enhance stealth.

A plug-and-play module that selects similar concept pairs and robust samples at the concept and instance levels, significantly improving the effectiveness of poisoning attacks such as AtoB and ShadowCast while maintaining model usability.

Demonstrates in medical question answering that retrieval systems are vulnerable to universal poisoning attacks, where attackers can inject documents containing target information such as PII, and proposes a detection-based defense.

Proposes an evaluation benchmark that tests the sensitivity of 22 models to data poisoning in the preference learning stage across 8 scenarios, finding that the poisoning effect has a log-linear relationship with the poisoning ratio and can generalize to unseen trigger words.

First knowledge poisoning attack against VLRAG systems, injecting just one poisoned sample into the knowledge base to manipulate answers to target queries, and extends to entire query classes.

Theoretically analyzes the stability conditions of self-consuming retraining loops under noisy and adversarial data curation, and designs an attack algorithm that lets competing platforms use malicious users to steer opponent models away from real user preferences.

Using inter-client consistency to estimate marginal contributions and reduce the impact of anomalous clients, then locally correcting suspicious gradients, defending against complex backdoor attacks without losing normal accuracy.

Finds that backdoored image representations are insensitive to changes in class description text, and accordingly proposes BDetCLIP, which detects CLIP backdoor samples at inference time using the similarity distribution difference between benign and malicious description texts.

For the first time quantifies the backdoor complications produced by backdoored pretrained language models in downstream tasks, and proposes a multi-task learning-based method that reduces complications while maintaining attack effectiveness.

Proposes a Data-Specific Indistinguishability framework that uses an optimal multivariate Gaussian mechanism to provide provable guarantees for memorization, data poisoning, and copyright, and validates its effectiveness in memorization mitigation, backdoor defense, and copyright protection.

First to study deception by untrusted NTL teachers in data-free distillation, proposing ATEsc to filter class-OOD synthetic samples based on adversarial robustness, effectively improving distillation results.

Proposes a single-run empirical audit method for f-DP, using the hypothesized f-DP curve to obtain tighter empirical privacy estimates than traditional (ε,δ).

Proposes the DSDA framework, which uses accelerated energy-guided data synthesis and discriminative-aware multi-task optimization to efficiently perform machine unlearning when the original training data is inaccessible, outperforming existing methods on three benchmarks.

Proposes AMUN, which fine-tunes on adversarial examples corresponding to forgotten samples to reduce confidence; after forgetting 10% of CIFAR-10 samples, even SOTA membership inference attacks do not outperform random guessing.

Identifies common weaknesses in adversarial training-based attribute removal methods, and proposes PASS with randomly replaced samples, validated on face, sensor, and speech data.

Points out that DP-SGD's formal guarantees do not apply to time series forecasting, analyzes the privacy amplification brought by structured subsampling, and implements training of forecasting models with strict event-level and user-level privacy guarantees.

Proposes certified machine unlearning using a surrogate dataset that approximates the source data distribution, calibrating noise by the statistical distance between the two, and still providing privacy guarantees without the original training data.

Proposes a method of public model decoding and private model safety verification, which accelerates safe inference decoding by 2.1 to 6.0 times while preserving privacy and generation quality.

For user data leakage in LLM inference, propose differential privacy prompt perturbation with a hybrid utility function and bucketed sampling, achieving a better privacy-utility trade-off.

Evaluates GNN vulnerability to model extraction attacks, proposes a strategy that iteratively optimizes node selection under banned batch queries, achieving better accuracy and fidelity than baselines under strict query budgets.

Uses convex relaxation and bound propagation to compute dataset-dependent upper bounds on prediction sensitivity, combined with the smooth sensitivity mechanism, making the privacy analysis of private predictions tighter by several orders of magnitude than global sensitivity methods.

Exploits the connection between forgetting and privacy amplification via random post-processing, achieving provable machine unlearning without loss function assumptions by fine-tuning with noise on retained data.

Proposes FedCEO, which performs tensor low-rank approximation on stacked local model parameters at the server side to recover semantics corrupted by noise, improving the utility-privacy trade-off bound by a factor of √d.

It proposes DP-TS-UCB, which uses a parameter α to trade off privacy and regret in differentially private stochastic bandits, providing GDP guarantees and regret upper bounds.

Proposes WASP, which uses a small number of private samples and weighted collaboration of multiple pretrained models to synthesize data under differential privacy guarantees, improving downstream task performance on 6 datasets.

Systematically evaluates voice cloning protection perturbations under realistic threats involving purification steps, proposes a two-stage purification method, and shows the limitations of adversarial perturbation-based defenses.

Under LabelDP constraints, optimizes the randomized response mechanism by maximizing the trace of the Fisher information matrix, achieving optimal estimation for binary response models, and provides private confidence intervals with coverage guarantees.

By querying a VLM with movie frames and analyzing text completions to infer whether training data contains copyrighted content, it releases the MovieTection benchmark; all tested models appear to have been exposed to such content.

Combines cryptographic protocols with distributed DP and matrix mechanisms to significantly improve federated learning model utility while supporting dynamic user exit, with minimal additional overhead.

For the first time proposes a machine unlearning framework TGU for zero-shot speech synthesis, enabling the model to no longer reproduce a specified speaker's voice while maintaining synthesis quality for other speakers.

For foundation models in split inference, using the prior of a pretrained latent diffusion model to reconstruct high-fidelity original images from deep intermediate representations, achieving better results than existing methods.

Proposes the POST framework, which tunes soft prompts locally on a small model (optionally with differential privacy) and then transfers them to a large model, avoiding sharing private data with the LLM provider and reducing cost.

Proposes the first first-person video privacy risk benchmark and retrieval-augmented attack, where foundation models can infer attributes such as wearer identity and gender with 70–80% accuracy in a zero-shot manner.

Finds that under the same (ε,δ)-DP guarantee, language models with different DP-SGD hyperparameter configurations differ significantly in empirical privacy as measured by memorization, and proposes a hyperparameter selection heuristic that considers empirical privacy.

Implements GPU-accelerated FHE and evaluates encrypted GPT-2 forward propagation, achieving over 200x speedup over the CPU baseline, while analyzing activation function approximations to preserve accuracy.

Using in-context learning plus quantized k-means to replace fine-tuning for adapting LLMs makes the cost of precisely forgetting fine-tuning data approximately constant, with performance comparable to fine-tuning approaches.

Proposes a threat model where the attacker only has partial, unordered sample information, and uses likelihood-ratio attacks and PRISM to infer sensitive fragments from fine-tuned LLMs, achieving performance comparable to data-aware baselines.

Proposes GI-DQA, which reconstructs private document content of document visual question answering models from shared gradients in federated learning, exposing privacy vulnerabilities in multimodal federated learning.

Studies user-level differentially private training when a single sample belongs to multiple users, gives definitions, a greedy baseline algorithm with contribution bounding, and analyzes its bias-variance trade-off.

We propose a dual-server secure aggregation scheme that resists model inconsistency attacks, supports verification of aggregation results, and reduces per-client overhead from logarithmic to constant.

Proposes FineXtract, which extrapolates and clusters the model distributions before and after fine-tuning to extract about 20% of the fine-tuning data from public fine-tuned diffusion models, revealing data leakage and copyright risks.

Uses per-sample privacy loss instead of worst-case bounds to quantify the difficulty of fine-tuning-based unlearning, improving the trade-off between utility and unlearning, and linking it to data difficulty metrics.

Studies how training randomness in data partitioning and model partitioning brings privacy amplification, and proposes balanced iteration subsampling, which in some cases provides stronger privacy amplification than Poisson sampling.

First unlearnable sample framework for multi-task data and MTL models, using a generator with label priors and intra-class embedding regularization to prevent unauthorized training, with stable attack effectiveness across 4 datasets.

Interpreting ICL as implicit Bayesian inference, it proposes PTA to modify the next-token distribution to generate DP synthetic examples, improving the accuracy of DP-ICL on text classification.

Privacy analysis of IAR shows that the new member inference attack achieves a TPR of 86.38% at 1% FPR, dataset inference requires only 6 samples, and hundreds of training images can be extracted, making it far more vulnerable than diffusion models.

Treats client private feedback as RL rewards, uses DPO to fine-tune LLMs to generate differentially private synthetic data; reduces the performance gap between fully private and non-private settings by 58%, outperforming existing synthetic data and DP-FL methods.

Studies personalized federated models with user-level differential privacy under a shared representation framework, proposing a private and efficient algorithm whose utility guarantees cover sub-Gaussian distributions and improve on existing privacy error terms.

Proposes the MaxAdaptiveDegree algorithm, which outputs more items under user-level differential privacy, scales to datasets with hundreds of billions of items, and outperforms existing parallel algorithms.

Establishes scaling laws that characterize DP LLM training properties, giving the overall relationship among computation, privacy, and utility, along with optimal training configurations under various settings.

Proposes the SecEmb protocol, which prevents the server from learning the item indices of user ratings and individual updates beyond the aggregated model, while reducing communication overhead by up to 90 times.

Generates multiple rewritten versions watermarked with a unique key, releases one and keeps the rest, and uses paired statistical tests comparing model likelihoods to detect whether data was used in pretraining; it can detect even when the data appears only once, at a proportion below 0.001%.

Without manually designing prompts or class names, using genetic algorithms to iteratively optimize prompts and synthesize images to steal black-box models, outperforming other methods under the same query budget.

Propose a system-aware unlearning definition that provides forgetting guarantees only against attackers who can access data stored by the system, and give an exact forgetting algorithm for linear classification along with a trade-off analysis.

Proposes the U-LiRA+ auditing method and the TULA attack, finding that text after unlearning can still be detected with high confidence, and that unlearning mechanisms instead increase membership inference and data reconstruction risks.

Design membership inference attacks against synthetic data, proving that synthetic text leaks fine-tuning data; and propose canaries with in-distribution prefixes and high-perplexity suffixes to improve privacy audit effectiveness.

Derives lower bounds on the success rate of membership inference attacks against fully connected or self-attention layers, showing that privacy risks remain under LDP protection, while the noise required to suppress attacks significantly harms utility.

Proposes a membership inference attack requiring only an image-to-image variant API, calling it multiple times and averaging outputs to compare with the original image, determining whether a sample belongs to the training set, and outperforming existing methods.

Proposes CafCor, which uses shared random seeds among participants to combine robust gradient aggregation with correlated noise injection, achieving a better privacy-utility trade-off than local differential privacy under malicious participants and an untrusted server.

Propose a minority-aware unlearning evaluation framework, using canaries containing PII to show that minority subsets have at least 20% higher privacy leakage under various unlearning methods and MIA.

Finds that VAE latent code variance affects image distortion, proposes LA and LE losses, enabling customized diffusion models to generate noise images with identity semantics completely erased, for forgery prevention.

By embedding task-irrelevant watermarks in VQA fine-tuning images and designing a probing framework, it finds that MLLMs unintentionally encode such private content, even when it does not affect outputs.

Infer the underlying GPU and software stack solely from the numerical patterns of a black-box LLM's inputs and outputs; under white-box conditions, GPU discrimination accuracy is 83.9%–100%, which can be used to verify whether a service provider is passing off cheap hardware or models as something else.

Using the freedom provided by quantization error, it constructs models that appear normal in full precision but behave maliciously after quantization; it achieves attacks such as unsafe code generation, content injection, and refusal on three models and nine GGUF types.

Built 7 ML research engineering environments and collected 71 expert attempts; under a 2-hour budget the best AI Agent scored 4 times that of humans, but under 8-hour and 32-hour budgets humans overtook it.

Uses password locking and circuit-breaking to train model organisms with hidden capabilities, finding that prompts can elicit MCQA capabilities, steering fails, and code tasks are only effective with fine-tuning.

A benchmark evaluating LLMs autonomously breaking adversarial sample defenses; the strongest agent combination can crack 87% of CTF-style defenses, but only 37% of real-world defenses.

Built a backend generation benchmark containing 392 tasks, using end-to-end exploits to test safety; the best model's functional correctness rate was only 62%, and about half of each model's correct programs could be successfully attacked.

Builds a sandbox benchmark based on high-risk CVEs to evaluate LLM agents' ability to exploit real web vulnerabilities; the most advanced agent framework can exploit up to 13% of the vulnerabilities.

Proposes a framework based on the Hessian condition number to analyze model immunity, designs a regularization algorithm that makes it difficult to fine-tune pretrained models on harmful tasks, and validates it on linear and deep models.

Using Self-Inf-N to select 100 outlier samples from benign data for fine-tuning can severely break the safety alignment of seven mainstream LLMs, and most existing mitigation measures fail.

Finds that existing defenses fail under large learning rates or multi-round fine-tuning, and proposes Antidote, which prunes harmful weights once after fine-tuning, reducing harmful scores while maintaining downstream accuracy.

Finds that some alignment data is more easily forgotten during harmful fine-tuning, and accordingly proposes VAA, which uses Group DRO for balanced learning and significantly reduces harmful scores across four types of fine-tuning tasks.

Demonstrates that leaderboards like Chatbot Arena can be attacked by identifying which model a reply belongs to and manipulating rankings with about a thousand votes, and collaborates with developers to implement mitigations.

Decomposing malicious tasks into subtasks and assigning each to the most suitable model, even if each individual model is safe, the combination can produce harmful content such as vulnerable code and hacker scripts at a higher rate.

Proposes GOAT, which automatically simulates multi-turn conversational attacks and combines 7 attack techniques, achieving ASR@10 of 96%, 91%, and 94% on Llama 3.1 8B, 70B, and GPT-4o respectively on JailbreakBench.

Trains an amortized investigator model that searches for prompts inducing target behaviors such as hallucinations and harmful responses, achieving a 100% attack success rate on AdvBench and an 85% hallucination induction rate.

Proposes the ELITE evaluator with toxicity scoring and a VLM safety benchmark, filters ambiguous low-quality samples from existing benchmarks and constructs diverse image-text combinations, with evaluation results more consistent with human judgment.

Based on in-context learning and using target VLM feedback, image-text adversarial test cases are generated, with both attack success rate and case diversity outperforming traditional red-teaming methods.