Constrained decoding: harmful or beneficial? Revisiting its impact on LLM performance

The study finds that constrained decoding does not necessarily harm LLM reasoning; its impact depends on the type of constraint. The authors divide constraints into three types—format, validity, and reasoning—and show that appropriate constraints can maintain or improve performance by eliminating invalid paths. Causal ablations on 4 model families and 11 open-source models show that apparent performance drops mainly stem from prompt-output format non-equivalence rather than decoder-level masking; validity constraints improve reliability, and reasoning constraints can improve reasoning and accuracy when enforcing beneficial trajectories.