Skip to content

RL-ATTACK: A General Black-Box Adversarial Attack on Language Models Based on Reinforcement Learning

RL-ATTACK: A Generalized Black-Box Adversarial Attack on Language Models via Reinforcement Learning

arXiv: Adversarial Attacks, Backdoors & Poisoning· arXiv:2611.08711· Taikun Chen, Yue Zhang, Wei Qian, Hui FangFinancial Services Analytics; University of Delaware; Department of Electrical and Computer Engineering, University of Delaware; Department of Applied Economics and Statistics, University of DelawareIndexed by this site · 11 days ago· Originally publishedAttacks

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors propose RL-ATTACK, which uses reinforcement learning to train a black-box attacker to generate adversarial perturbations against a target model; on SNLI it reduces BERT's post-attack accuracy to 2.03% within 13 queries.

Threat modelBlack-box attack: the attacker cannot access the target model's gradients and can only obtain prediction confidence (soft-label) or labels (hard-label baseline) through queries. The NLI task goal is to make the classifier misclassify; the generative Q&A goal is jailbreak, making the target LLM output judged unsafe by the safety evaluation. The attacker controls token substitutions in the input document; the victim systems are a BERT classifier and Llama 2 7B, GPT-3.5-Turbo, GPT-5.4-mini.

Questions
Black-box adversarial attacks cannot access the target model's gradients; existing methods mostly rely on external knowledge to guess transformations, ignoring target model feedback, making it difficult to customize attacks for a specific model.
Methods
RL-ATTACK uses Policy Gradient to train an attacker based on a BERT masked language model, with the decline in the target model's confidence as the reward; it first identifies high-impact tokens and then replaces them with sampled candidates, adapting to both classification and generation tasks.
Experiments and results
On SNLI/MNLI, RL-ATTACK reduces BERT's post-attack accuracy to 0.80%–2.03% with 13–16 queries; on AdvBench it achieves 11.5% post-attack accuracy against GPT-3.5-Turbo, with USE higher than PAIR.
Limitations and future work
The authors note that the conclusions are limited by the scope of the evaluation setting, the small size of the evaluation set, and the lack of explicit syntactic constraints; more advanced models such as GPT-5 were not evaluated due to the risk of account bans.
Experimental setupBERT、Llama 2 7B et al. · SNLI、MNLI (matched/mismatched) et al. · Post-attack Accuracy、Number of Queries et al.
Evaluated models
BERTLlama 2 7BGPT-3.5-TurboGPT-5.4-miniLlama 3 8B Instruct
Benchmarks
SNLIMNLI (matched/mismatched)in-the-wild jailbreak-prompt corpusAdvBench
Metrics
Post-attack AccuracyNumber of QueriesUSEPercentage of Perturbation
AI summaryResearchers propose RL-ATTACK, a reinforcement learning-based black-box adversarial attack framework that learns from target model feedback to generate targeted adversarial perturbations rather than relying on external knowledge to guess transformations. On natural language inference tasks, the method reduces post-attack accuracy to 2% within 20 queries; for generative language models, the trained attacker consistently achieves lower post-attack accuracy than untrained attackers and wins 4 of 6 comparisons against strong optimization baselines, while also being superior in query efficiency and semantic preservation. The authors also test multiple reward designs and training settings and find that none is consistently dominant. Code is open-sourced at https://github.com/taikunudel/RL-ATTACK。

Researchers propose RL-ATTACK, a reinforcement learning-based black-box adversarial attack framework that learns from target model feedback to generate targeted adversarial perturbations rather than relying on external knowledge to guess transformations. On natural language inference tasks, the method reduces post-attack accuracy to 2% within 20 queries; for generative language models, the trained attacker consistently achieves lower post-attack accuracy than untrained attackers and wins 4 of 6 comparisons against strong optimization baselines, while also being superior in query efficiency and semantic preservation. The authors also test multiple reward designs and training settings and find that none is consistently dominant. Code is open-sourced at https://github.com/taikunudel/RL-ATTACK。

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Existing black-box attacks use static frameworks to guess transformations, ignoring target model feedback; the authors propose a trainable RL black-box attack framework.

    Core problem: Existing black-box adversarial attacks cannot access the target model's gradients and usually rely on external knowledge to guess adversarial transformations, ignoring the information provided by the target model, making it difficult to customize attacks for a specific model.

    • Problem setting and importance: Language models (such as BERT classifiers and generative models like Llama and GPT) are susceptible to adversarial perturbations, producing incorrect or unsafe outputs. Black-box attacks can identify vulnerabilities without gradients, which has practical value for improving model safety.
    • Shortcomings of existing methods: Classifier attacks such as TextFooler and BERT-ATTACK use static frameworks that apply the same strategy to all models, missing opportunities to customize attacks for a specific model; among generative model attacks, GCG is a white-box method, and although PAIR is black-box, the authors believe its semantic preservation and query efficiency have room for improvement.
    • Core observation: Reinforcement learning's reward signal can learn from target model feedback in a gradient-free environment, guiding the attacker to generate adversarial transformations specific to a given model.
    • Paper's approach: Proposes RL-ATTACK, a trainable black-box attack framework based on Policy Gradient, adaptable to natural language inference (NLI) classification tasks and generative Q&A tasks.
    • Threat model:
      • Attacker's goal: For NLI, to flip the classifier's prediction to any incorrect label; for generative Q&A, to induce the target LLM to produce a response judged unsafe by the safety evaluation (jailbreak).
      • Knowledge: Black-box setting; the attacker cannot access the target model's gradients and can only obtain prediction confidence (soft-label) or labels (hard-label baseline) through queries.
      • Capability: The attacker controls token substitutions in the input document (the edit budget K limits the number of perturbed tokens per document).
      • Victim systems: BERT classifier for the NLI task; Llama 2 7B, GPT-3.5-Turbo, GPT-5.4-mini for the generative Q&A task.
  2. What related research exists?

    Related work falls into three categories: classifier black-box attacks, generative model attacks, and reinforcement learning applications; baselines include TextFooler, BERT-ATTACK and PAIR.

    Black-box attacks against language classifiers

    • TextFooler [11] — constructs a candidate set (based on semantic similarity) and applies syntactic/semantic filters for token replacement.
    • Generative Attack [10] — generates candidates using GloVe embeddings and a language model filter to maintain fluency.
    • BERT-ATTACK [8] — uses a masked language model to propose context-aware replacement words.
    • Other hard-label attacks — SSPAttack [27], HQA-Attack [28], TextHoaxer [29], LeapAttack [30], etc., which use only predicted labels rather than confidence scores.

    Attacks against generative language models

    • Greedy Coordinate Gradient (GCG) [16] — a white-box attack that trains a universal adversarial suffix appended to benign prompts to induce harmful outputs.
    • PAIR [17] — a black-box attack that uses a language model to interactively paraphrase adversarial prompts; the authors use it as a strong optimization baseline for comparison.

    Applications of reinforcement learning in NLP

    • Policy Gradient [18] — the training method adopted in this paper.
    • RL for sequence generation — e.g., summarization and translation tasks using ROUGE, BLEU, or human preference scores as rewards [20]–[25].

    Baselines and benchmarks

    • Baseline methods: For the NLI task, comparison against TextFooler, BERT-ATTACK, and four hard-label attacks; for generative Q&A, comparison against PAIR and an untrained BERT attacker.
    • Benchmarks/datasets: SNLI, MNLI (matched/mismatched), in-the-wild jailbreak-prompt corpus, AdvBench.

    The authors position this paper's distinction from existing work: existing black-box attacks lack the ability to customize for a specific target model, whereas RL-ATTACK learns from target model feedback via reinforcement learning, achieving a trainable, model-specific black-box attack.

  3. How does the paper address this problem?

    RL-ATTACK uses Policy Gradient to train a BERT masked language model attacker, with the decline in target confidence as the reward, identifying high-impact tokens and then replacing them.

    RL-ATTACK is a trainable black-box attack framework based on reinforcement learning; its core idea is to train an attacker to learn from target model feedback to generate adversarial transformations specific to a given model, rather than using static heuristic rules.

    Problem formulation

    Given a source document xi = [ti1, ..., tim], the attacker Aθ generates an adversarial document x′i in which only tokens at a few positions S(xi) are replaced. The attacker is built on a pretrained encoder (BERT), and the top layer Tθ outputs a probability distribution over the vocabulary via softmax(W h + b), used to sample adversarial token candidates.

    Training objective and reward design

    • Adversarial reward: The default variant A is the raw decline in the target model's confidence for the true label, C(ytrue|x) − C(ytrue|x′); variant B uses a sigmoid-bounded reshaping; variant C weights by the original confidence (Table I).
    • Semantic preservation reward: Uses Universal Sentence Encoder (USE) to compute the cosine similarity rsem between the original and perturbed sentences.
    • Combined reward: r = α · radv + (1−α) · rsem, where α controls the trade-off between adversarial effectiveness and semantic preservation.

    Attacker training

    Policy Gradient is used to optimize the expected reward Rθ. The gradient is estimated using the likelihood ratio trick and Monte Carlo sampling, with token substitutions performed only at target positions S(xi). During training, perturbation positions are chosen randomly, encouraging the attacker to learn effective substitution strategies independent of position.

    Attack mechanism

    1. Identify high-impact tokens: Compute each token's impact score I(tn) = C(ytrue|x) − C(ytrue|x∖ tn), and select the top-K ranked tokens as the target set S(x) (K is the edit budget hyperparameter).
    2. Iterative attack: For each position in S(x), take the top-N candidates from the trained attacker's distribution, greedily replace and keep the candidate that most reduces the true-class probability, until the prediction changes or the maximum number of queries is reached.

    Task adaptation and success criteria

    • NLI task: The target is the classifier, the reward is the decline in true-label confidence, and attack success is defined as the prediction flipping to any incorrect label.
    • Generative Q&A: The target is the LLM; during training, Llama-Guard-4-12B is used as a safety evaluator to provide confidence (the target LLM is not queried to reduce training cost); at attack time, the adversarial prompt is submitted to the target LLM, and Llama-Guard-4-12B or the OpenAI Moderation API judges whether the response is unsafe; success means the evaluation judges it unsafe.
  4. What experiments did the paper conduct?

    The experiments cover NLI (SNLI/MNLI) and generative Q&A (AdvBench); RL-ATTACK achieves 0.80%–2.03% post-attack accuracy on NLI with 13–16 queries.

    Experimental setup

    • Models evaluated: BERT for the NLI task; Llama 2 7B, GPT-3.5-Turbo, GPT-5.4-mini for generative Q&A.
    • Datasets: SNLI, MNLI (matched/mismatched), in-the-wild jailbreak-prompt corpus (16,776 train/50 traindev/100 evaldev), AdvBench (500 entries).
    • Baselines: For NLI, compared with TextFooler, BERT-ATTACK, SSPAttack, HQA-Attack, TextHoaxer, LeapAttack; for generative, compared with PAIR and an untrained BERT attacker.
    • Metrics: Post-attack Accuracy (proportion still correct after attack, lower is better), Number of Queries (average number of queries), USE (semantic similarity, higher is better), Percentage of Perturbation (proportion of perturbed tokens).
    • Evaluation method: The generative task uses Llama-Guard-4-12B (training and main evaluation) and the OpenAI Moderation API (supplementary evaluation) to judge whether a response is unsafe.
    • Number of repetitions: Both NLI and generative results report standard errors from 10-fold cross-validation.

    Main results

    According to Table II (NLI) and Table III (generative Q&A, AdvBench):

    Swipe horizontally to read the table

    TaskTarget modelMethodPost-attack Accuracy# QueriesUSE% Perturbation
    SNLIBERTRL-ATTACK2.03% (±0.54)130.655.1
    SNLIBERTBERT (untrained)5.73%180.606.0
    SNLIBERTTextFooler4.00%600.4518.5
    MNLI matchedBERTRL-ATTACK0.90% (±0.30)150.703.2
    MNLI mismatchedBERTRL-ATTACK0.80% (±0.28)160.733.0
    AdvBenchLlama 2 7BRL-ATTACK82.0% (±3.8)27.70.982.0%
    AdvBenchGPT-3.5-TurboRL-ATTACK11.5% (±3.3)16.80.984.5%
    AdvBenchGPT-5.4-miniRL-ATTACK92.6% (±2.6)29.00.991.1%
    AdvBenchGPT-3.5-TurboPAIR28.0%29.60.44N/A
    • The authors state that RL-ATTACK achieves the lowest post-attack accuracy among soft-label attacks on all three NLI datasets, with far fewer queries (13–16) than hard-label baselines (1000).
    • On generative tasks, RL-ATTACK achieves lower post-attack accuracy than PAIR in four of six settings, with USE (0.96–0.99) higher than PAIR (0.44–0.60).
    • Exceptions: GPT-5.4-mini under Llama Guard evaluation has RL-ATTACK at 92.6%, higher than PAIR's 77.6%; GPT-3.5-Turbo under Moderation API evaluation has RL-ATTACK at 42.0%, slightly higher than PAIR's 40.0%.

    Hyperparameter sensitivity and training effects

    • Grid search: 99 configurations tested on traindev (reward variants A/B/C × α ∈ {0.0,...,1.0} × K ∈ {3,5,10}), with post-attack accuracy ranging from 0.64–0.80 (mean 0.71, standard deviation 0.035), and no consistently optimal reward type or hyperparameter (Figure 3).
    • Training effects: Comparing untrained attackers with selected checkpoints, post-attack evaluation accuracy decreases in all six representative configurations (e.g., 0.16→0.15 to 0.18→0.02), but traindev accuracy cannot reliably rank held-out performance (Figure 4).

    Other ablations and analyses

    • Edit budget K: Fixed at K=3 for the NLI task; the authors state that a small edit budget helps semantic preservation.
    • Candidate budget N: N=40 at attack time; the authors observe diminishing returns as N increases.
    • Evaluator model generalization: The OpenAI Moderation API is used as a supplementary evaluator to verify the generalization of an attacker trained on Llama Guard under different safety models.
  5. What are some points that can be further explored?

    The authors note that the conclusions are limited by the scope of evaluation, the size of the evaluation sets, and the lack of grammaticality constraints; more advanced models were not evaluated.

    Limitations noted by the authors and future directions

    • Limited evaluation scope: The conclusions are limited by the scope of the evaluation settings (Conclusion).
    • Evaluation set size: The evaluation sets are of modest sizes (Conclusion).
    • Lack of grammaticality constraints: There is a lack of an explicit grammaticality constraint on adversarial documents (Conclusion).
    • Model coverage: More advanced proprietary models (such as GPT-5, Claude, Gemini Pro) were not evaluated, because providers deploy automatic account-banning mechanisms, making systematic evaluation infeasible (Section IV-C).

    Scope of experimental coverage

    • Models evaluated: NLI uses only BERT; generative Q&A uses Llama 2 7B, GPT-3.5-Turbo, GPT-5.4-mini, for a total of 4 target models.
    • Defense evaluation: The paper does not report evaluation against defense methods.
    • Sample sizes: NLI uses standard evaluation sets; in generative Q&A, AdvBench has 500 entries, the in-the-wild training set has 16,776 entries, traindev has 50 entries, and evaldev has 100 entries.
    • Number of repetitions: Main results report standard errors from 10-fold cross-validation.
    • Unreported information: The paper does not report a specific upper limit on the query budget when attacking generative targets (only average query counts are reported); it does not report agreement between human evaluation and automatic evaluation.
  6. Summarize the main content of the paper

    RL-ATTACK is a trainable black-box attack framework based on reinforcement learning, achieving model-specific attacks with low queries and high semantic preservation on NLI and generative Q&A.

    RL-ATTACK is a trainable black-box adversarial attack framework based on reinforcement learning, aimed at solving the problem that existing black-box attacks cannot be customized for a specific target model. The framework uses Policy Gradient to train an attacker based on a BERT masked language model, with the decline in the target model's confidence as the reward signal; it first identifies high-impact tokens and then replaces them with sampled candidates, without needing access to the target model's gradients.

    On NLI tasks, RL-ATTACK reduces BERT's post-attack accuracy to 2.03%, 0.90% and 0.80% on SNLI, MNLI matched and MNLI mismatched, respectively, with an average of only 13–16 queries, a perturbation rate of about 3%–5%, and a USE semantic similarity of 0.65–0.73, outperforming soft-label baselines such as TextFooler and BERT-ATTACK. On generative Q&A tasks, against Llama 2 7B, GPT-3.5-Turbo and GPT-5.4-mini, RL-ATTACK achieves lower post-attack accuracy than PAIR in four of six evaluation settings, with USE scores (0.96–0.99) higher than PAIR (0.44–0.60) and fewer queries. Against GPT-3.5-Turbo under Llama Guard evaluation, it achieves 11.5% post-attack accuracy.

    The authors experimented with multiple reward designs and training hyperparameters and found no consistently optimal configuration, so they adopted a simple setting (reward variant A, α=1, K=3). The authors state that reinforcement learning training generally improves attack effectiveness without sacrificing semantic preservation, but the conclusions are limited by the scope of evaluation, the small size of the evaluation sets, and the lack of explicit grammaticality constraints, and more advanced models such as GPT-5 were not evaluated.

Related papers

Show 3 more papers
Read originalarxiv.org