Skip to content

Hardening agent network populations via multi-criteria competitive co-evolution: insights from wargame strategy optimization

Hardening Agentic Web Populations Through Multi-Criterion Competitive Co-Evolution: Lessons from Wargame Strategy Optimization

arXiv: 其他 AI 安全论文· arXiv:2611.19319· Shashank Raj, Kalyanmoy DebDepartment of Computer Science, The University of AlabamaIndexed by this site · 2 days ago· Originally publishedDefenses· Its place in the scan record →

Quick read

Other

Based on the paper PDF (AI-generated); refer to the original

The authors put forward the position of transferring the multi-criterion competitive coevolution mechanism from wargaming to the adversarial contest between jailbreak attacks and safety guardrails, and list open questions for deployment on the agentic web.

Questions
Existing language model safety evaluations are mostly one-off static red teaming: fixed jailbreak prompts against fixed models, which cannot characterize the ongoing arms race in which attackers and defenders continually adapt to each other; the authors argue this means defenses will be found and broken by subsequent attacks.
Methods
Position paper: maps the dual-population competitive coevolution mechanism in wargame strategy optimization (alternating evolution, cross-play evaluation matrix, NSGA-II, Pareto front) onto a jailbreaker population and a guardrail population, and borrows the Movement metric (HVcoop − HVcoev) to measure the competitive pressure borne by both sides.
Experiments and results
The paper did no experiments; the authors argue the mechanism naturally fits the multi-criterion trade-offs of language model safety, and point to three reasons at web scale—discovery pairing, credit assignment and trust propagation—as well as open questions including evaluation cost and multiple-population alternation.
Limitations and future work
The authors note: the cross-play matrix requires Noff × Ndef evaluations per generation, and language model calls are costly, possibly requiring surrogate models or opponent subsampling; whether the alternating loop remains effective under multiple populations and different update cadences is unknown; the method carries dual-use risk, and the authors state that access to the attacker population should be controlled in implementation.
Experimental setupMovement(HVcoop − HVcoev)、hypervolume
Threat model
The attacker side is a continually adapting population of jailbreakers (candidate prompts, templates or strategies), whose goal is to elicit prohibited responses while remaining hard to detect; the defender side is a population of guardrails (system prompts, classifiers or refusal strategies), whose goal is to intercept prohibited responses while maintaining a low false-refusal rate. The victim system is a population of language model agents interacting at large scale on the open web.
Metrics
Movement(HVcoop − HVcoev)hypervolume
AI summaryThe paper proposes having jailbreak attacks and defenses and guardrails co-evolve adversarially as two populations, rather than a fixed attacker probing a fixed defense, in order to strengthen agent security on open networks. The idea originates from wargame strategy optimization: two-population competitive coevolution with alternating evolution and cross-play evaluation, which has been shown to produce strategies robust to a moving opponent. The authors point out that real applications need to balance multiple conflicting criteria such as cost and efficiency, and argue that this is a networked population-level problem rather than a two-agent toy setting.

The paper proposes having jailbreak attacks and defenses and guardrails co-evolve adversarially as two populations, rather than a fixed attacker probing a fixed defense, in order to strengthen agent security on open networks. The idea originates from wargame strategy optimization: two-population competitive coevolution with alternating evolution and cross-play evaluation, which has been shown to produce strategies robust to a moving opponent. The authors point out that real applications need to balance multiple conflicting criteria such as cost and efficiency, and argue that this is a networked population-level problem rather than a two-agent toy setting.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors argue one-off static red teaming cannot characterize the attack-defense arms race, and advocate two-population competitive coevolution to harden agentic web systems.

    Core problem: language model safety evaluation remains one-off static red teaming and cannot characterize the ongoing mutual adaptation of attackers and defenders; the authors argue a defense tested only once will eventually be found and broken.

    • Problem setting: on the agentic web, attackers who fail against today's guardrails keep adjusting until a new prompt works, while a defender that has only seen yesterday's attacks is blind to tomorrow's; static red teaming benchmarks freeze one side of the interaction and cannot capture this back-and-forth.
    • Shortcomings of existing methods: the authors say existing adversarial robustness work typically evaluates fixed models with a fixed set of jailbreak techniques (Perez et al., 2022; Zou et al., 2023; Chao et al., 2025), reflecting only one snapshot of attack and defense; a defense trained only against the current strongest attack will overfit, and vice versa. The authors also say existing coevolutionary research either does not consider multiple objectives for each agent, or combines multiple criteria into one using subjective, non-general scalarization.
    • Core observation: wargame strategy optimization has already used two-population competitive coevolution (alternating evolution + cross-play evaluation) to produce strategies robust to a moving opponent (Guha et al., 2024, 2025a,b), and the same mechanism can also measure competitive pressure with the Movement metric (Raj et al., 2026a,b); the authors believe it can be transferred directly to language model safety.
    • Paper's contribution: the authors state explicitly that the contribution is a position rather than a new algorithm—arguing that the mechanism is suited to hardening agentic web systems, and describing what must change in transferring from a wargame simulator to web-scale language model agent populations.
    • Threat model:
      • Attacker goal: elicit prohibited responses while remaining hard to detect.
      • Defender goal: intercept prohibited responses while maintaining a low false-refusal rate.
      • Forms of both sides: the attacker side is a population of jailbreak strategies (candidate prompts, templates or prompting strategies), and the defender side is a population of guardrail strategies (system prompts, classifiers or refusal strategies).
      • Victim system: a population of language model agents interacting at large scale and discovering each other on the open web.
  2. What related research exists?

    Related work falls into three groups: wargaming coevolutionary optimization, competitive balance measurement, and language model red teaming evaluations; the authors position this paper as a position on mechanism transfer.

    Competitive coevolutionary optimization of wargame strategies

    • Guha et al. (2024)—formalizes attack and defense strategy optimization as a two-population multi-criterion problem: each strategy plays against all members of the opposing population (cross-play matrix), fitness is aggregated over the whole row/column, and the two populations evolve in turn with NSGA-II on an alternation period [τoff, τdef].
    • Guha et al. (2025a)—adds progressive shrinking of the decision space to this framework, targeting more realistic sequential decision-making.
    • Guha et al. (2025b)—adds a regularity-based procedure to select one executable strategy from the Pareto front.

    Measuring competitive balance (companion work to the same mechanism)

    • Raj et al. (2026a) and Raj et al. (2026b)—use the same two-population alternating cross-play mechanism to measure game balance: comparing the hypervolume under the cooperative upper bound with the hypervolume actually reached at the coevolutionary equilibrium, the difference being Movement; if the Movement on both sides is roughly equal, the interaction is judged balanced. Raj et al. (2026b) also removes the surrogate model and operates directly on game dynamics to avoid surrogate error.

    Adversarial robustness evaluation of language models

    • Perez et al. (2022), Zou et al. (2023), Chao et al. (2025)—the authors say such work evaluates fixed models with a fixed set of jailbreak techniques, which can find real weaknesses but reflects only one snapshot of attack and defense; this is the comparison target named in the paper.

    Survey of coevolutionary multi-objective algorithms

    • Antonio and Coello (2017)—the paper cites its conclusion that overfitting to a single opponent is a known failure mode of competitive coevolution, and that cross-play aggregation is precisely aimed at it.

    Baselines and benchmarks: the paper is a position paper, with no experiments and no comparison baselines or benchmark datasets.

    Authors' positioning: the authors say the paper proposes no new algorithm, but argues that the wargaming version of the mechanism is a natural choice for hardening agentic web systems, and lists open questions between transferring from a simulator to web-scale agent populations; the multi-criterion, non-scalarized Pareto strategy set is what the authors emphasize as the difference from existing coevolutionary research.

  3. How does the paper address this problem?

    Directly maps the two-population multi-criterion competitive coevolution mechanism from wargaming to alternating adversarial evolution between a jailbreaker population and a guardrail population.

    Overall idea: transfer the multi-criteria competitive coevolution mechanism from wargame strategy optimization (Figure 1) to language model safety—the jailbreaker population and the guardrail population score each other and evolve alternately, each maintaining its own Pareto-optimal strategy set, with the goal that both sides be robust to the coevolved opponents (rather than fixed opponents).

    Problem formalization

    • Objective vector: one match between attack strategy x and defense strategy y produces f(x,y) = [f1off(x,y), f2off(x,y), f1def(x,y), f2def(x,y)]T, i.e., a vector of two objectives for each side.
    • Evaluation tensor: an attack population of size Noff and a defense population of size Ndef play all pairwise matches, yielding at generation t an evaluation tensor Ft ∈ R^{Noff × Ndef × 4}.
    • Fitness aggregation: each strategy's fitness is aggregated over all opponents it faces, rather than looking at the result against a single opponent; the authors state this step prevents strategies from winning by overfitting to a single opponent.

    Alternating evolution and multi-criteria optimization

    • Alternation cycle: with period [τoff, τdef], one side evolves for τoff generations with NSGA-II while the other is frozen, then the roles swap; the authors state that alternation plus cross-play aggregation makes the Pareto front represent strategies robust to the adapting opponents.
    • Pareto strategy set: each side keeps its own Pareto-optimal solution set; the authors argue that multi-criteria lets the agent choose among multiple trade-off strategies at each step, whereas single-criterion methods produce only one strategy per step and can only "accept or discard".

    Mapping to jailbreak and defense

    • Attack population: candidate prompts, prompt templates, or prompting strategies, with the goal of eliciting prohibited responses and being hard to detect.
    • Defense population: system prompts, classifiers, or refusal strategies, with the goal of intercepting prohibited responses while keeping the false-refusal rate low.
    • Why multi-criteria is needed: the authors state that an attacker that always succeeds but is easily detected is not strong, and a defender that never fails but rejects half of benign requests is useless, so a single win/loss signal cannot be used.

    Movement diagnostic metric

    • Definition: Movement = HVcoop − HVcoev, i.e., the difference between the hypervolume under the cooperative upper bound and the actual hypervolume at the coevolutionary equilibrium, quantifying how much one side is forced to sacrifice due to competitive pressure.
    • Upper bounds in the safety context: for the jailbreaker side, the cooperative upper bound is its attack success rate against an undefended model; for the defender side, the cooperative upper bound is its refusal accuracy against fully cooperative, non-adversarial users.
    • Use: the authors state that this difference can directly measure "how much cost a given guardrail imposes on attackers, and how much usability a given attacker costs defenders", providing transparent trade-off information to both sides at each step.

    Envisioned extension to the agentic web

    • Discovery and pairing: at web scale, attackers and defenders are heterogeneous, independently operated, mutually discovering agent populations, and who pairs with whom and how pairings evolve become part of the problem.
    • Trust propagation: the authors hope that an agent robust to multiple coevolved manipulation behaviors can carry this reputation into interactions with unfamiliar agents, corresponding to the manipulation-resistant reputation problem raised at the workshop.
  4. What experiments did the paper conduct?

    The paper is a position paper and did no experiments; it only restates existing practice of the transferred mechanism in the wargaming domain.

    Experimental setup

    • The paper ran no experiments: the full text did not run any model, benchmark, or simulation, and has no experimental setup, tested models, datasets, metric evaluations, or review method.
    • Figure 1 is a mechanism diagram: the jailbreaker population and the defender population score each other in a cross-play matrix, fitness vectors are aggregated across all opponents, the two populations evolve in turn with the other frozen, and each maintains its own Pareto-optimal solution set; the figure contains no numerical results.

    Main results

    • The paper has no results tables or experimental figures, and no numbers to report.
    • The authors' core claims are given in argument form: cross-play aggregation prevents overfitting to a single opponent, alternating evolution prevents the two populations from drifting apart, and the multi-criteria Pareto set allows choosing among multiple trade-off strategies at each step.

    Reported conclusions of prior work (not experiments in this paper)

    • Source of the mechanism: the authors report that Guha et al. (2024, 2025a,b) used this mechanism to optimize attack and defense strategies in wargame simulation, and Raj et al. (2026a,b) used the same mechanism to measure game balance; these conclusions come from the cited work and were not reproduced in this paper.
    • Surrogate model trade-off: the authors report that the earliest wargame version used surrogate models (Guha et al., 2024), and newer versions removed surrogates entirely and operate directly on game dynamics to avoid surrogate error (Raj et al., 2026b); the authors consider language model calls not cheap, and this trade-off needs to be re-studied for the agentic web.

    Other ablations and analyses

    • No ablations, parameter analyses, or negative results to report.
  5. What are some points that can be further explored?

    The authors list open questions including evaluation cost, multiple-population alternation and dual use; the paper did no experiments, and its coverage is empty.

    Limitations and future directions noted by the authors

    • Evaluation cost (Section 5): the cross-play matrix requires Noff × Ndef evaluations per generation, and each evaluation is a language model call rather than a cheap simulation step, so cost grows quickly; the authors suggest surrogate modeling or opponent subsampling may be needed, and the trade-off between surrogate accuracy and evaluation cost needs to be re-studied for the agentic web.
    • Multiple-population alternation (Section 5): a more realistic agentic web has multiple attack and defense lineages rather than two populations; the authors say it is not yet known whether the alternating loop that keeps two populations from drifting still works when multiple populations update at different rhythms.
    • Agent ordering and cooperation/conflict (Section 5): how agent ordering affects the final optimal strategy in multi-agent settings needs study; the effect of cooperation or conflict among multiple jailbreaker agents on the defender's optimal strategy is another direction.
    • Convergence (Section 4): the authors say whether alternating cross-play coevolution still converges rather than destabilizing as population size and the number of simultaneously evolving lineages grow is an open question.
    • Dual use (Section 5): a method that coevolves strong jailbreak strategies can by construction also produce strong jailbreak strategies; the authors say any implementation should restrict access to the coevolved attacker population and report only defender robustness rather than attacker prompts.
    • Future plans (Section 5): the authors plan to embed this range of possibilities in an agentic web application.

    Scope of experimental coverage

    • The paper ran no experiments: no tested models, datasets, baselines or evaluation protocols.
    • The paper reports no quantitative results; all mechanism descriptions are restated from cited work in the wargaming domain (Guha et al., 2024, 2025a,b; Raj et al., 2026a,b).
    • The paper gives no concrete implementation parameters for transfer to language model agents (population size, alternation period and objective function instantiation are all not instantiated).
  6. Summarize the main content of the paper

    Position paper: advocates using the wargaming multi-criterion competitive coevolution mechanism to have jailbreaker and guardrail populations evolve adversarially, and lists open questions for deployment on the agentic web.

    A position paper at the NeurIPS 2026 Agentic Web workshop, arguing for transferring the two-population multi-criterion competitive coevolution mechanism from wargame strategy optimization to language model safety.

    • Problem to solve: the authors argue one-off static red teaming freezes one side of the attack-defense interaction and cannot characterize the arms race in which attackers and defenders continually adapt to each other on the agentic web, and a defense tested only once will eventually be broken.
    • Method highlights: a jailbreaker population (prompts, templates or strategies) and a guardrail population (system prompts, classifiers or refusal strategies) play pairwise in a cross-play matrix, fitness is aggregated across all opponents, the two populations evolve in turn with NSGA-II on an alternation period, each maintaining a Pareto-optimal strategy set; the Movement metric (the difference between the cooperative upper-bound hypervolume and the hypervolume at equilibrium) is borrowed to measure the competitive pressure borne by each side.
    • Key claims (in the authors' voice): cross-play aggregation prevents overfitting to a single opponent, alternating evolution prevents population drift, and multiple criteria let an agent choose among multiple trade-off strategies at each step rather than "accept or forgo" a single strategy.
    • Results: the paper did no experiments and has no quantitative results; the argument is based on restating existing work in the wargaming domain (Guha et al., 2024, 2025a,b; Raj et al., 2026a,b).
    • Authors' conclusion: extending the mechanism to web-scale heterogeneous, discoverable agent populations would turn it from an adversarial training technique into a genuine agentic web research problem; evaluation cost, the effectiveness of multiple-population alternation, and dual-use control are open questions that must be resolved before deployment.
Read originalarxiv.org