Eliciting Language Model Behaviors with Investigator Agents

1. 01What problem does this paper attempt to solve?

   The paper attempts to solve how to automatically, at scale, and diversely elicit specific target behaviors (such as jailbreaks, hallucinations, and abnormal behaviors) from target language models.

   The paper attempts to solve how to automatically and diversely elicit specific output behaviors from target language models (behavior elicitation).

   • Application scenarios and importance: Model developers train models with behavior guidelines or constitutions to ensure good deployment behavior, but models still often exhibit unexpected anomalous behaviors due to the vast behavior space under free-text input.
   • Shortcomings of existing methods: Human red teaming is costly and hard to scale; existing automated jailbreak methods (e.g., GCG) are mostly limited to narrow task distributions, require time-consuming per-instance search, and often generate unreadable gibberish prompts.
   • Core observation and hypothesis: The authors frame behavior elicitation as amortized posterior inference, hypothesizing that training a general investigator agent on a high-level target distribution can shift the combinatorial search cost in input space to the training phase and discover diverse prompt strategies expressed in natural language.
   • Threat model:
     • Attack goal: elicit the target language model to output specific target strings (e.g., harmful content) or responses satisfying natural language evaluation criteria (e.g., hallucination, abnormal psychological behaviors).
     • Attacker knowledge: during training, needs to query the target model's generation results and output log probabilities (to compute elicitation reward and fluency cross-entropy); during transfer evaluation, does not access internal parameters of non-target models.
     • Attacker capability: the attacker controls single-turn input prompts (maximum length 128 tokens), samples 25 candidates from the investigator model, re-ranks based on elicitation log probabilities, and submits to the target model.
     • Victim system: the victim system is a pretrained or aligned language model; main experiments evaluate Llama-3.1 8B and Llama-3.1 8B-Instruct.

   Collapse this answer

2. 02What related research exists?

   The paper discusses three major categories of related work: automated red teaming and jailbreaks, amortized red teaming and language model inversion, and diversity and reinforcement learning optimization.

   • Automated red teaming and jailbreaks
     • GCG (Zou et al., 2023) and AutoDAN (Liu et al., 2024) — model jailbreaking as prompt optimization for a single sample at inference time, with high search cost, and GCG often generates gibberish token sequences.
     • Adaptive Attacks (Andriushchenko et al., 2024) — break safety-aligned models using longer prefix prompts, but the generated prompts are long.
     • Multi-turn human interaction (Li et al., 2024; Ayrey, 2024) — discover various anomalous behaviors through open-ended human interaction with LLMs, but it is time-consuming and hard to scale.
   • Amortized red teaming and language model inversion
     • Reinforcement learning red teaming (Perez et al., 2022; Hong et al., 2024) — train models via reinforcement learning to perform red teaming, amortizing search cost; the authors note that it mainly induces broad harmful behaviors, while this paper supports fine-grained behaviors and conditioning on evaluation criteria.
     • Language model inversion (Pfau et al., 2023; Donahue et al., 2020; Morris et al., 2023) — Pfau et al. pretrain models that invert to predict tokens; Donahue et al. fine-tune models to fill in blanks; Morris et al. recover prompts based on next-token probabilities. The authors note that this paper inverts specific target models and introduces a DPO reward, supporting inversion of strings not generated by the model and generalization to criteria.
   • Preference optimization and diversity reinforcement learning
     • DPO (Rafailov et al., 2023) and mode collapse (Song et al., 2023) — direct preference optimization algorithms often suffer from mode collapse due to preference learning, causing the model to degenerate into a single policy with repetitive suffixes.
     • Frank-Wolfe algorithm (Frank & Wolfe, 1956; Bach, 2012) — a classic conditional gradient optimization method; the authors borrow its principles to apply adaptive regularization penalties to prompts discovered in previous rounds.
   • Baseline methods and benchmark datasets
     • Baselines: SFT baseline, standard single-round DPO baseline, GCG (cited from Zou et al., 2023), and Adaptive Attacks (cited from Andriushchenko et al., 2024).
     • Benchmarks and datasets: AdvBench (including Harmful Strings and the 50-task Harmful Behaviors subset), FineWeb, TruthfulQA, DSM-5 diagnostic manual, WildChat, and UltraChat.
   • Positioning relative to existing work: The authors state that this paper unifies behavior elicitation as amortized posterior inference, enabling not only exact string matching but also elicitation against natural language evaluation criteria, and achieves diverse exploration of readable policies by decoupling diversity and fluency regularization.

   Collapse this answer

3. 03How does the paper address this problem?

   The paper formalizes behavior elicitation as amortized posterior inference and proposes a multi-stage reinforcement learning training framework combining SFT, DPO, and Frank-Wolfe iterations.

   The authors propose training an Investigator Agent to amortize the input search cost into the training phase, combining supervised fine-tuning (SFT), direct preference optimization (DPO), and Frank-Wolfe (FW) diversity regularization for behavior elicitation.

   Core objective and formal definition

   In the string elicitation task, given a target output y, the investigator model p_θ(x ∣ y) generates a prefix prompt x to maximize the log probability of the target model generating y. To balance elicitation success rate, diversity, and readability, the optimization objective is defined as: L(p_θ) = R(p_θ) + β₁ H(p_θ) − β₂ H(p_θ, p_m) where R(p_θ) = 𝔼_{y ∼ P, x ∼ pθ(· ∣ y)}[log p_m(y ∣ x)] is the elicitation reward, H(p_θ) is the prompt entropy (encouraging policy diversity), and H(p_θ, p_m) = 𝔼[−log p_m(x)] is the cross-entropy (constraining prompt fluency under the target model).

   Multi-stage training pipeline for string elicitation

   1. Supervised fine-tuning (SFT) initialization: Leveraging the asymmetry that forward generation is easier than backward inversion, sample x from the prefix distribution and have the target model greedily decode to generate suffix y, building a dataset to fine-tune the investigator model to maximize log p_θ(x ∣ y), bringing generated prompts into the correct semantic space.
   2. DPO preference fine-tuning: To improve elicitation reliability, sample prompt pairs from the current investigator model, construct preference pairs (x^w, x^l) based on the target model's elicitation log probabilities, and use the DPO loss for on-policy iterative fine-tuning.

   Frank-Wolfe diversity optimization

   Standard DPO is prone to mode collapse (e.g., degenerating into a single policy with repetitive suffixes). The authors decouple the optimization objective into a smooth function term and a KL divergence term, deriving an iterative algorithm equivalent to Frank-Wolfe optimization:

   • In the i-th iteration, the DPO reward function for optimizing model q⁽ⁱ⁾ penalizes prompts already discovered by the aggregated model from previous rounds: r(x, y) = log p_m(y ∣ x) − λ log p_θ⁽ⁱ⁻¹⁾(x ∣ y), where λ = β₁ − β₂.
   • Then update the aggregated model with a step size: p_θ⁽ⁱ⁾(x ∣ y) = (1 − η_i) p_θ⁽ⁱ⁻¹⁾(x ∣ y) + η_i q⁽ⁱ⁾(x ∣ y). The first round is equivalent to standard DPO; subsequent rounds force the model to explore different prompt strategies such as prefixed linearized tables, natural continuations, etc.

   Extension to criteria-based behavior elicitation

   For open-ended natural language criteria (R, a), the authors decompose it into a two-stage concatenated string elicitation problem:

   1. Stage one (derive response y from criteria): Train a model q(y ∣ R, a), initialized with SFT, then optimize with the FW-DPO algorithm using the verifier LLM's scoring log probability log p_v(a ∣ R, y) as reward.
   2. Stage two (derive prompt x from response y): Reuse the SFT model trained for string elicitation, and run the FW algorithm on the response set sampled from stage one, using the target model's elicitation score log p_m(y ∣ x) as reward to generate prompt x.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The paper evaluates on pretrained strings, harmful jailbreaks, hallucinations, and abnormal behaviors, achieving elicitation success rates above 80% on multiple tasks (according to Tables 4 and 7).

   Experimental Setup

   • Models under test: The main experiments tested Llama-3.1 8B and Llama-3.1 8B-Instruct; cross-model transfer experiments evaluated Llama-3.3 70B-Turbo, GPT-4o (with both system and user prompt conditions), and Claude 3.5 Sonnet; model scale ablations tested Llama-3.2 1B as the investigator base.
   • Datasets and scale: Pretraining Strings (4096 samples sampled from FineWeb), AdvBench Harmful Strings (574 samples), AdvBench Harmful Behaviors (50-task subset), TruthfulQA (3600 constructed hallucination criteria), DSM-5 manual (3804 constructed aberrant behavior criteria).
   • Baseline methods: SFT baseline, single-round DPO (FW-1) baseline; jailbreak tasks compared against GCG (Zou et al., 2023) and Adaptive Attacks (Andriushchenko et al., 2024).
   • Metric definitions: ASR (attack success rate), ASR@25 (best success rate among 25 candidate samples), elicited log probability (log p_m(y ∣ x), including mean and best scores), entropy, verifier score (log p_v(a ∣ R, y)).
   • Evaluation method: Harmful Behaviors used the GPT-4 automatic judge proposed by Chao et al. (2023), combined with Llama-3.1 8B as the training-time harmfulness judge; hallucination tasks used Llama-3.1 8B-Instruct as the verifier; aberrant behaviors were judged by GPT-4o mini; at inference, k=25 candidates were sampled from the investigator model and reranked based on elicited log probability.
   • Repetition and compute setup: Experiments were run on a single node with 8×A100 or 8×H100 GPUs; FW iterations were typically 3 to 4 rounds.

   Main Results

   The main experiments' performance across different task settings is as follows (per Table 2, Table 4, Table 7):

   Swipe horizontally to read the table

   Task / Dataset	Target Model	Method Variant	Main Metric Value	Entropy / Verifier Score	Source
   Pretraining Strings (n=4096)	Llama-3.1 8B	SFT	Best Elicit: -0.844	Entropy: 0.897	Table 2
   Pretraining Strings (n=4096)	Llama-3.1 8B	FW-1 (DPO)	Best Elicit: -0.621	Entropy: 0.400	Table 2
   Pretraining Strings (n=4096)	Llama-3.1 8B	FW-full	Best Elicit: -0.528	Entropy: 1.432	Table 2
   AdvBench Harmful Strings (n=574)	Llama-3.1 8B	Ours (DPO)	ASR: 98.8%	Not reported	Table 4
   AdvBench Harmful Behaviors (n=50)	Llama-3.1 8B	Ours (DPO)	ASR: 100%	Not reported	Table 4
   Hallucinations (n=3600)	Llama-3.1 8B-Instruct	FW-full	ASR@25: 85.2%	Verifier: -2.612	Table 7
   Aberrant Behaviors (n=3804)	Llama-3.1 8B-Instruct	FW-full	ASR@25: 80.7%	Verifier: -0.798	Table 7

   • Elicitation quality vs. diversity trade-off: The authors state that on the pretraining strings test, DPO improved decoding scores but reduced entropy (from 0.897 to 0.400), while FW-full raised entropy to 1.432 while achieving a best elicitation score of -0.528.
   • Jailbreak success rates outperform traditional baselines: On the AdvBench benchmark, against Llama-3.1 8B, the DPO investigator achieved ASRs of 98.8% and 100% on Harmful Strings and Harmful Behaviors, respectively, higher than the 57.0% and 56% reported for GCG in the literature (the authors note that GCG was tested on Llama 2 with different splits, so the comparison is not fully comparable).
   • Criteria elicitation performance: On hallucination and aberrant behavior tasks, FW-full achieved ASR@25 of 85.2% and 80.7%, respectively, both higher than SFT (81.7% and 75.5%) and single-round DPO (83.7% and 80.6%).

   Cross-Model Transfer Experiments

   • Evaluation setup: Attack prompts (best-of-25) generated by a DPO investigator (FW-1) trained on Llama-3.1 8B were input to non-target models not involved in training (Table 5).
   • Experimental results: On the Harmful Behaviors task, transfer to Llama-3.3 70B-Turbo achieved an ASR of 98%, GPT-4o (system) 28%, Claude 3.5 Sonnet 26%, and GPT-4o (user) 16%; on the Harmful Strings task, Llama-3.3 70B-Turbo achieved 75.8%, Claude 3.5 Sonnet 11.9%, GPT-4o (system) 10.6%, and GPT-4o (user) 9.4%.
   • Authors' interpretation: The authors state that the attack prompts transfer at a high rate to a larger model in the same family (Llama-3.3 70B-Turbo) and partially transfer to closed-source commercial models.

   SFT Dataset Selection Ablation

   • Evaluation setup: Compared the impact on jailbreak capability of SFT initialization using real user conversation data (WildChat) versus purely synthetic instruction data (UltraChat) (right column of Table 5).
   • Experimental results: Investigators trained on UltraChat maintained 100% ASR on Harmful Behaviors against Llama-3.1 8B, achieved a transfer ASR of 88% to Llama-3.3 70B-Turbo, but transfer ASR dropped to 0% for GPT-4o (system) and Claude 3.5 Sonnet, and 8% for GPT-4o (user).
   • Authors' interpretation: The authors state that this result shows that even without fine-tuning on datasets containing real human attack examples, training on synthetic data alone can achieve high elicitation success rates against the target Llama models.

   Investigator Model Scale Ablation

   • Evaluation setup: Used the smaller Llama-3.2 1B as the investigator base model, trained with up to 15 rounds of DPO against Llama-3.1 8B (Table 8).
   • Experimental results: The 1B investigator reached 100% ASR on Harmful Strings after 10 rounds of DPO (log probability -0.0016) and 84% ASR on Harmful Responses after 7 rounds of DPO (log probability -0.0019).
   • Authors' interpretation: The authors note that an investigator model with 1B parameters still possesses strong capability and can effectively elicit responses from the more powerful 8B target model.

   Other Ablations and Analyses

   • DPO iteration count analysis: On the harmful strings test, elicited log probability first rose then fell with training; too many iterations caused entropy to drop sharply and sampled gibberish (Figure 5).
   • FW iteration count analysis: On pretraining string elicitation, the FW algorithm tended to converge after about 4 iterations; the second iteration caused a temporary drop in elicitation score due to penalizing the previous policy, but entropy improved noticeably (Figure 4).
   • Step-size update mechanism ablation: Using uniform averaging η_i = 1/i compared to the classical convex optimization step size η_i = 2/(i+1) ultimately converged to a better objective value L and higher diversity score (Figure 6).
   • Single-stage vs. two-stage DPO comparison: On in-distribution criteria elicitation, two-stage DPO achieved a verifier score of -0.70, better than single-stage DPO's -1.13 and SFT's -1.13 (Table 9).
   • Policy diversity coverage: Three FW iterations covered different levels of text repetition rates, and the aggregated distribution Pareto-dominated SFT and DPO on both elicited log probability and repetition rate (Figure 7).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out limitations such as reward hacking of LLM verifiers; future work could explore tool augmentation and multi-turn reasoning elicitation mechanisms.

   Limitations and future directions noted by the authors

   • Verifier risk of reward hacking: The authors note that using language models for verification carries a risk of reward hacking, and instances of reward hacking against LM judges were observed in experiments (Section 5.2 footnote 1, Section 7 Limitations).
   • Need for more robust verification mechanisms: The authors propose future systematic analysis of reward hacking and exploration of enhancing verifiers with tools or external databases (Section 7 Limitations).
   • Single-turn limitation and multi-turn elicitation extension: The authors note that this paper only explores the single-turn dialogue setting; future work could extend to multi-turn scenarios, allowing the investigator to submit multi-turn prompts and leverage inference-time computation (Section 7 Discussion).
   • Incorporating tool use and chain-of-thought reasoning: The authors propose giving the investigator the ability to use tools (e.g., querying other models or accessing internal model representations) and chain-of-thought reasoning to discover more complex real-world behaviors and narrow the gap with human red teams (Section 7 Discussion).

   Experimental coverage

   • Scope of target models tested: The main experiments target Llama-3.1 8B and Llama-3.1 8B-Instruct; transfer experiments evaluate Llama-3.3 70B-Turbo, GPT-4o, and Claude 3.5 Sonnet, totaling 5 models.
   • Types of tasks and behaviors covered: Experiments cover pretrained string reconstruction, AdvBench harmful strings and the 50-task harmful behaviors subset, TruthfulQA hallucination criteria, and DSM-5 abnormal psychological behavior criteria.
   • Input format and interaction rounds: All experiments are limited to single-turn plain-text prompt inputs, with a fixed maximum prompt length of 128 tokens.
   • Judgment and verification methods: Harmful behaviors rely on GPT-4 automatic scorers and Llama-3.1 8B judges; criteria tasks rely on Llama-3.1 8B-Instruct or GPT-4o mini for judgment; the paper does not report agreement between automatic judgments and human annotations.
   • Sample sampling and evaluation scale: In the evaluation phase, 25 candidate prompts are always sampled from the investigator model for re-ranking; the harmful behaviors subset test sample size is 50.

   Collapse this answer

6. 06Summarize the main content of the paper

   The paper proposes using reinforcement learning to train an investigator agent to automatically induce multiple types of behaviors in large models, balancing induction effectiveness and policy diversity.

   • Core positioning and problem: Addressing the challenges that complex language model behaviors are difficult to characterize comprehensively, traditional red-teaming search is expensive, and prompts lack readability, the paper proposes an automated behavior elicitation framework based on an Investigator Agent.
   • Method design highlights: The authors model behavior elicitation as amortized posterior inference, proposing a multi-stage RL pipeline comprising forward-data SFT initialization, DPO fine-tuning based on elicited log probability, and an adaptive diversity regularization algorithm based on the Frank-Wolfe conditional gradient idea; for natural language criteria, they propose a two-stage process that first derives responses and then derives prompts.
   • Main experimental results: On Llama-3.1 8B, the method achieved ASRs of 98.8% and 100% on AdvBench Harmful Strings and the 50-task Harmful Behaviors subset, respectively, and prompts transferred to Llama-3.3 70B-Turbo with an ASR of 98%; on TruthfulQA hallucination elicitation and DSM-5 aberrant psychological behavior elicitation, ASR@25 reached 85.2% and 80.7%, respectively.
   • Strategy discovery and ablation performance: The FW iterative algorithm successfully discovered diverse strategies including leading tables and natural continuations, mitigating the mode collapse problem of standard DPO while maintaining high elicitation scores; ablation experiments showed that a small investigator model with 1B parameters can also effectively elicit responses from an 8B target model.
   • Authors' conclusions and implications: The authors believe that the single-round investigator model demonstrates the feasibility of amortized behavior elicitation, and that extending the investigator to an agent system supporting multi-round interaction, tool use, and multi-step reasoning in the future may further narrow the gap between automated methods and human expert red teams.

   Collapse this answer

Related papers