Self-organizing learning in decentralized agent collectives: market-based post-training (MPT)

Decentralized agent populations need to learn both how to act and when and by whom to participate; fixed orchestration trains these two sides separately.

• Scenario: The authors argue that population capability depends on both members and how they are coordinated. Existing systems mostly place models into predetermined structures: managers delegate, verifiers check generators, or they communicate according to fixed workflows (Hong et al., 2024; Du et al., 2024; Li et al., 2023; Wu et al., 2024). Even when participants are trainable, the orchestration deciding who acts next is often kept fixed during training (Feng et al., 2026).
• Coupling: Who acts in a given state both changes the trajectory and determines who receives experience and learning signals. As individuals become stronger, effective participation changes: agents that should act early may later become unnecessary; after downstream agents strengthen, previously unhelpful actions may become valuable.
• Existing shortcomings: Training under fixed orchestration optimizes only one side of this coupling. The authors frame the difficulty as joint orchestration and credit assignment: each round decides who acts; after feedback arrives, it must also decide how to train the agents that shaped the trajectory. Participation determines who gets experience, and learning changes who should participate later.
• This paper's approach: The authors ask whether a decentralized population can learn both action and self-orchestration from experience. market-based post-training (MPT) lets agents express participation willingness via bids; bids determine who acts, and collective outcomes provide local credit for actions and bids, with no central coordination agent. Experimental tasks are embodied crafting, robot navigation, and deep research.

In Section 4, the authors divide related work into three groups and name orchestration baselines in the experiments.

LLM multi-agent systems

• Roles and workflows: Generative Agents (Park et al., 2023) studies social interactions with memory and reflection; CAMEL (Li et al., 2023) uses role-playing to study cooperation. MetaGPT and ChatDev (Hong et al., 2024; Qian et al., 2024) organize software development with structured workflows and specialized roles; AutoGen and AgentVerse (Wu et al., 2024; Chen et al., 2024) provide programmable conversation and collaborative teams; multi-agent debate (Du et al., 2024) improves reasoning through iterative exchange and revision of answers.
• Inference-time interaction adjustment: GPTSwarm (Zhuge et al., 2024) optimizes graph connections; AgentNet (Yang et al., 2025) uses experience and evolutionary connections for decentralized routing; Graph-of-Agents (Yun et al., 2026) selects agents and refines answers through graph-based collaboration. The authors note that these systems generally adjust interactions at inference time, while MPT learns policies through weight updates.

Multi-agent reinforcement learning

• Classic credit and communication: MADDPG and MAPPO (Lowe et al., 2017; Yu et al., 2022) use centralized training with decentralized execution; VDN and QMIX (Sunehag et al., 2018; Rashid et al., 2020) decompose team value into individual utilities; COMA (Foerster et al., 2018) uses counterfactual baselines for credit assignment. RIAL and DIAL (Foerster et al., 2016), CommNet (Sukhbaatar et al., 2016), and IC3Net (Singh et al., 2019) learn communication or when to communicate.
• LLM-oriented training: Multiagent Finetuning (Subramaniam et al., 2025) updates specialized models with their own interaction data; Dr. MAS (Feng et al., 2026) uses per-agent advantage normalization to stabilize reinforcement learning; CoLLM (Liu et al., 2026) studies centralized and decentralized critics. The authors note that these methods do not connect "who is the next agent" with "how downstream rewards are distributed among agents."

Economic mechanisms in agents

• Markets and credit along chains: Hayek (1945) discusses prices conveying dispersed knowledge; Minsky (1986) describes intelligence as the organization and interaction of finite components. Contract Net (Smith, 1988) allocates tasks through negotiation; Wellman (1993) uses markets for decentralized resource allocation. Holland (1995)'s bucket brigade passes credit along chains of active rules; Baum (1996; 1999)'s Hayek Machine develops economic credit assignment.
• Frozen-weight LLM auctions: Economy of Minds (Qi et al., 2026) uses auctions to coordinate LLM agents and evolves prompts and populations under frozen weights; SALE (Alazraki et al., 2026) uses cost–value scoring and shared auction memory to select among strategies proposed by agents. The authors note that recent LLM-based economic systems mainly evolve prompts with feedback while keeping weights frozen; MPT uses feedback to train action and bidding policies.

Baselines and benchmarks

• Baselines and tasks: The training baseline is single-agent REINFORCE with discount γ=0.95. Inference-time orchestration baselines are Round-Robin (Wu et al., 2024), AgentNet (Yang et al., 2025), and Graph-of-Agents (Yun et al., 2026). Tasks are embodied crafting in MineStudio (Iron Pickaxe, and later Torch, Stone Bricks), VLN-CE R2R in Habitat, and OpenResearcher.

The authors position MPT as: using learned bids to connect both turn allocation and credit assignment, thereby improving participation and policy together without a central coordinator; and distinguishing it from methods that only change interactions at inference time, and from frozen-weight methods that only evolve prompts.

MPT lets each agent use the same set of parameters to produce both actions and bids: the bid determines who gets the action right and how much of the collective reward is passed back to earlier participants. There is no central scheduler. The auction mechanism follows Qi et al. (2026), but here task feedback updates the weights.

Episodes, populations, and round allocation

• Setup: Episodic tasks with verifiable rewards. An episode gives task x, unfolding over rounds t=1,…,T; at most one agent acts per round, and a verifier gives task reward R. The population has N agents with mutually independent parameters, and policy πi generates both actions and bids.
• Auction: Each round, agents independently submit bids bi based on local context. The highest bidder acts, and its bid is recorded as bt. Ties are broken uniformly at random among the highest bidders. If all bids are 0, no one acts that round and it is skipped.
• Cold start: Supervised fine-tuning is done before post-training. In MPT, each agent is fine-tuned only on its role demonstrations; the single-agent baseline is trained on all role demonstrations, and the total SFT compute on both sides is matched.
• Discrete bid levels: The bid prompt in Appendix B restricts bids to five levels: 0, 0.2, 0.4, 0.6, 0.8; 0 means abstaining. The credit formula in the main text still treats bt as the pass-back proportion. Figure 2 illustrates with Iron Pickaxe, e.g., Smith wins with 0.8.

Using bids for credit assignment and learning actions

• Pass-back rule: The environment reward at round t is denoted r̂t, with the terminal verified reward included in r̂T. Cumulative value and local credit are defined as v_t=r̂_t+b_t+1v_t+1, r̃_t=(1−b_t)v_t, with vT+1=0. The actor keeps proportion 1−bt as its own action credit and passes proportion bt back to the previous actor; the share that the first actor continues to pass back is discarded.
• Downstream weights: The weight with which reward at round t+k enters vt is the product of bj from j=t+1 to t+k. The authors say this follows the bucket brigade (Holland, 1995; Baum, 1999), letting downstream outcomes provide signals to earlier actions. Figure 3 illustrates forward action and backward credit assignment.
• Parameter updates: Each agent updates only with the action tokens from rounds in which it acted, using REINFORCE (Williams, 1992). The baseline is that agent's average credit over those rounds in the batch. A higher bid increases the chance of acting while committing a larger share of outcome value to earlier participants.

Learning bids with counterfactual comparisons

• Contested rounds: If the second-highest bid is nonzero, the round is marked contested. The original branch is the collected trajectory. Figure 4 illustrates comparing round allocations caused by different bids from the same state.
• Alternative branch: In a copy, the environment and agent states of that round are restored, and the selected agent's bid is resampled at most 3 times until it is below the second-highest bid, so the second-highest bidder acts instead. If no such bid is obtained, the round is excluded from bid training. The alternative branch continues to the end of the episode.
• Preference pairs: Each branch is scored by the sum of environment rewards from the contested round onward, not by the passed-back local credit. The bid response with the higher score is yw, the other yl; if the two branch returns are equal, the pair is discarded. The Bradley–Terry objective increases the relative likelihood of the bid corresponding to the better outcome. Appendix Algorithm 1 gives the full procedure.

Alternating updates

• Schedule: 4 action updates are followed by 1 bid update. In Algorithm 1, when the iteration index k modulo 5 is not 0, actions are updated; otherwise, bids are updated.
• Same parameters: Both updates optimize the same set of parameters within that agent, so action capability and participation tendency change together. The paper does not give a bid loss coefficient other than the learning rate in the main text; the post-training learning rate is in Appendix B.

Experimental setup

• Models and roles: crafting uses three independently fully fine-tuned Qwen3-VL-4B-Instruct models (Logger, Miner, Smith), with language instructions given to frozen STEVE-1. navigation uses three NaVILA-8B models (Turner, Mover, Stopper), with action space divided into turn, move, and stop. research uses three gemma-4-E2B-it models (Searcher, Opener, Finder), with tools divided into search, open, and in-page find; any agent can give the final answer. The single-agent baseline is REINFORCE on the same base model, γ=0.95.
• Environments and rewards: crafting is in MineStudio, starting from an empty backpack to an iron pickaxe, with 120 decisions per episode and 40 frames per instruction; stage rewards for twelve tech-tree events sum to a maximum of 140. navigation is on VLN-CE R2R in Habitat, with at most 64 decisions; stopping within 3 m of the goal counts as success, with distance coefficient λd=1.0 and success reward λs=2.5. research uses OpenResearcher, with at most 64 decisions and terminal reward 0 or 1.
• Data: crafting post-training uses 1614 world seeds; SFT uses 4791 samples (Logger 1677, Miner 2380, Smith 734), generated by Claude Sonnet 5. navigation post-training uses 10744 R2R-CE training episodes distinct from SFT; SFT retains 1931 successful decisions, labeled by Claude Sonnet 5. research post-training uses 5000 questions distinct from SFT; SFT uses 568 assistant turns from the dataset. Post-training learning rates are all 1×10−6, train batch sizes are 32, 16, and 64 respectively, and sampling temperature is 1.0 (Appendix B, Tables 3–5).
• held-out: disjoint from training. crafting uses 32 unseen world seeds; navigation uses 88 R2R-CE val-unseen episodes; research uses 50 held-out questions. Each instance is evaluated 3 times under the training setup. Table 1 reports mean ± standard error over instances, with each instance first averaged over the three runs.
• Metrics and evaluation: crafting and navigation use average episode reward; research uses average accuracy. For research, correctness is judged by qwen3.5-flash using OpenResearcher's HLE template against reference answers, giving yes/no. The paper does not report agreement between this judge and humans, nor statistical significance tests.
• Compute comparison: Figure 5 compares under matched training FLOPs. FLOPs count all LLM computation in training, including bids, actions, counterfactual rollouts, and parameter updates for all N agents.

Main results

Setting	Single-agent RL	MPT
crafting, Qwen3-VL-4B-Instruct, average reward, Table 1	20.3±4.1	41.5±5.9
navigation, NaVILA-8B, average reward, Table 1	0.52±0.14	1.23±0.18
research, gemma-4-E2B-it, average accuracy, Table 1	9.0±2.1%	21.0±4.1%

• held-out point estimates: In the table, MPT point estimates are higher than single-agent RL for all three. The authors report MPT is higher than single-agent RL on all three.
• Training curves are not the same numbers: The main text's reading of Figure 5 is that at matched FLOPs end, crafting is about 41 vs 20, navigation about 0.9 vs 0.3, and research about 27% vs about 6%. These are training curve readings, separate from Table 1's held-out. The authors claim MPT is higher than single-agent post-training under matched training compute.

How participation changes, and orchestration baselines

• Participation shares: Figure 6 compares on Iron Pickaxe after SFT, before RL, two RL intermediate checkpoints, and after RL. The authors state that at decision step 120, Logger goes from about 80% before RL to 45% after RL, Miner from about 10% to 40%, and Smith from about 10% to 15%. They treat this as behavioral evidence that action turn allocation changes during training.
• Fixed action policy: Table 2 left keeps the post-training action policy fixed and compares orchestration on the same 32 seeds. Round-Robin is 23.8±4.2, AgentNet is 20.9±4.2, GoA is 25.4±4.6, and MPT is 41.5±5.9. The authors claim a 63.4% relative improvement over the strongest baseline GoA, and state that learned self-orchestration better coordinates these agents. This 41.5±5.9 is the same as the crafting row in Table 1; the paper does not state whether these 32 seeds are the held-out seeds.
• Both components trained: Table 2 right crosses initial and post-training checkpoints on the same 32 seeds. Initial bids plus initial actions is 15.4±4.1; training only bids is 21.5±4.7; training only actions is 27.2±4.8; training both is 41.5±5.9. The authors claim the full training configuration is 2.69 times the initial configuration, and argue that effective self-orchestration requires both strong actions and bids.

New goals, inference-time populations, and dropout

• Continued training: Starting from their respective Iron Pickaxe checkpoints, they continue training on Torch and Stone Bricks separately. The main text's reading of Figure 8 is that at about 10^18 adaptation FLOPs, the population is about 35 and 38, while the single agent is 17 and 19. The authors consider this supports reusing trained populations to speed up adaptation.
• Replication rather than retraining: Populations of 3, 6, 9, and 12 are obtained by replicating each trained agent 1 to 4 times, with no additional training. On Iron Pickaxe, 32 seeds, 3 runs per seed, they report pass@3 for five subtasks. The authors state that going from 3 to 12, Wooden Pickaxe goes from about 78% to 97%, Stone Pickaxe from 69% to 94%, and Furnace from 66% to 91%, and claim all five subtasks improve. The main text does not give specific pass@3 for Iron Ingot and Iron Pickaxe.
• Dropout: At inference time, agent dropout is introduced with dropout probability p; the paper does not explain how a single agent is made unavailable. The authors' reading of Figure 10 is that at p=0.9, the 12-agent population has average episode reward about 33, while the 3-agent population has 19. The authors claim extra members provide redundancy, helping maintain capability when members are unavailable.

Other ablations and analyses

• Round-robin training (Figure 7): Bids are removed, turns rotate, and reward discount γ=0.95. At about 5×10^18 FLOPs, the main text reading is average reward about 15, while MPT is about 42. The authors claim round-robin improves little during training. This is not the same setting as the inference-time Round-Robin in Table 2.
• Random credit (Figure 7): Turn allocation and bid optimization remain unchanged, but the bids used for credit assignment are replaced with U(0,1). At the same FLOPs reading, it is about 31. The authors claim learned bids also provide more effective action training signals.
• Single-episode walkthrough (Appendix C): One each for crafting, navigation, and research, not used as aggregate metrics. The crafting case obtains the iron pickaxe at decision 74, with that event reward being +30.

Limitations and future directions noted by the authors

• External rewards: MPT relies on externally evaluated task rewards to train actions and bids, and its effectiveness depends on the quality of these signals. The authors state they have not evaluated settings where outcomes are largely subjective or where usable feedback is hard to obtain (Appendix D).
• Restorable states: Bid updates require counterfactual rollouts from the same environment and agent states. The authors say extending to irreversible real-world interactions remains an open problem (Appendix D).
• Model and population scale: The authors say computational constraints limited the study's scale; they have not explored populations of multiple models each over 10B parameters, nor jointly trained much larger agent populations (Appendix D).

Experimental coverage

• Three tasks, three backbones: embodied crafting, robot navigation, and deep research each use 3 role-complementary, parameter-independent agents, with backbones Qwen3-VL-4B-Instruct, NaVILA-8B, and gemma-4-E2B-it (§3.1). The single-agent control is REINFORCE on the same backbone, γ=0.95.
• Held-out and repetitions: 32 unseen world seeds, 88 R2R-CE val-unseen episodes, 50 held-out questions, 3 runs per instance (Table 1). The correctness-judging model for research is only stated as qwen3.5-flash.
• Tasks for orchestration and ablations: Round-Robin, AgentNet, GoA, and bid/action checkpoint combinations are all compared on Iron Pickaxe's 32 seeds (Table 2). Round-robin training and random credit ablations are also on Iron Pickaxe (Figure 7).
• Post-training usage: Torch and Stone Bricks continue training from Iron Pickaxe checkpoints (Figure 8). At inference, populations of 3, 6, 9, 12 are obtained by copying trained agents; dropout experiments are also on Iron Pickaxe, 32 seeds, 3 runs per seed (Figure 9, Figure 10).
• Quantities not reported in the paper: statistical significance tests, agreement between research reviewers and humans, and the proportion of contested rounds discarded due to resampling failure or equal branch returns.

Decentralized groups need not first fix who acts when. MPT uses sealed bids simultaneously as an action-right auction and a credit return ratio: the highest bidder acts, and accumulated value is distributed to earlier participants according to that bid; actions are updated with REINFORCE, bids are learned with preference learning on counterfactual branches over the same state, with 1 bid update after every 4 action updates.

• held-out: on crafting, MPT's average reward is 41.5±5.9, single-agent RL is 20.3±4.1; on navigation, 1.23±0.18 vs 0.52±0.14; on research, accuracy is 21.0±4.1% vs 9.0±2.1% (Table 1, 3 runs per instance). Training curves are compared by matched training FLOPs, and their readings are not the same set of numbers as this held-out table.
• Orchestration and components: with the action policy fixed, on Iron Pickaxe MPT is 41.5±5.9 and GoA is 25.4±4.6; the authors claim a relative improvement of 63.4% (Table 2). Training only bids or only actions is lower than training both together; the authors claim full training is 2.69 times the initial configuration.
• Use after training: the authors claim the group adapts faster when moving from Iron Pickaxe to Torch and Stone Bricks. When replicating the trained agents to 12 without additional training, pass@3 on Wooden Pickaxe goes from about 78% to 97%. Under p=0.9 dropout, the authors' reading of Figure 10 is about 33 for a 12-agent group and 19 for a 3-agent group.
• The authors' conclusion: the authors argue that self-orchestration can become a learnable, reusable component, allowing individual capabilities and participation style to change together with experience. Appendix D also states that this depends on externally scoreable rewards and counterfactual replay over recoverable states.