LambdaMix: Controllable multi-objective group-relative preference optimization based on jointly trained low-rank adapters

The paper proposes a parameter-efficient post-training method, LambdaMix, which uses group-relative online RL to learn a family of policies conditioned on preferences. The method jointly trains one low-rank adapter for each objective on a frozen base model; a single preference vector both selects the mixture of adapters used to generate responses and weights their respective normalized rewards for updating the adapters. After one training run, preferences can be switched at inference time without further optimization. The authors tested on four base LLMs across three settings: helpfulness-harmlessness trade-off, math and instruction following, and rewriting reviews while removing personal attribute cues. They report that LambdaMix's hypervolume is higher than all evaluated conditional and policy-merging baselines, and that it uses fewer training completions and adapter storage than a set of GRPO-specific models. Code is open-sourced at https://github.com/ckarouzos/lambdamix。