CaRe: Correcting Modality Misrouting in Full-Modality LLMs to Mitigate Cross-Modal Hallucinations

Under audio-visual interference, omni-modal large models can still answer correctly from the question-relevant unimodal branch, but the joint prediction does not use that evidence.

• Scenario: The authors state that OLLMs process audio, visual, and text simultaneously, but joint modeling does not guarantee faithful grounding. Ungrounded audio-visual hallucinations limit video understanding, audio-visual QA, and embodied applications.
• Existing shortcomings: The authors state that existing mitigations mainly rely on training-time alignment, representation intervention, or training-free decoding correction to suppress hallucinations or recalibrate modality contributions; cases where relevant unimodal evidence is recoverable but not incorporated into the joint prediction are less targeted.
• Observation: The authors call the gap between erroneous joint predictions and recoverable unimodal predictions modality misrouting. On erroneous joint predictions, the target branch recovery rate is 52.9%–74.4% on CMM and 37.3%–46.2% on AVHBench (Figure 3; the text gives ranges without mapping each to specific models and subsets).
• Method: The authors propose a training-free inference-time method CaRe (Conflict-aware Residual Intervention): after detecting candidate misrouting, it uses counterfactual residuals to steer the fused representation toward the selected modality while preserving the original audio-visual paths.
• Threat model:
  • Failure object: Joint audio-visual reasoning of open-source omni-modal models. The paper posits no external adversary; interference comes from competing modalities.
  • Failure form: Evidence for the modality required by the question is still readable from the unimodal branch, but the joint prediction leans toward another modality. Figure 2 shows two directions: visible objects inducing false sounds, and sounds inducing false visual objects.
  • Method permissions: The model is frozen, parameters unchanged; it can withhold the competing modality for counterfactual forward passes and modify decoder hidden states of final prompt tokens.
  • Non-trigger: When the probe does not uniquely point to audio or video, or when the two branches have no hard decision conflict, the original joint prediction is preserved.

Omni-modal models

• OneLLM (Han et al., 2024), VITA (Fu et al., 2026), Qwen-Omni and MiniCPM-o (Xu et al., 2025a; Yao et al., 2025; Team, 2026; Xu et al., 2025b; Cui et al., 2026), OmniVinci (Ye et al., 2026): extend language models to unified processing of audio, visual, and text.
• Park et al. (2025), Zhang et al. (2025): the authors cite them to illustrate that competing or imbalanced modalities may prevent question-relevant information from entering the joint prediction. The paper does not frame this as a criticism of these two works themselves.

Hallucination evaluation

• CMM (Leng et al., 2026) and AVHBench (Sung-Bin et al., 2025): target multimodal over-reliance and cross-modal hallucination, respectively. The main experiments use these two benchmarks.
• Zechen et al. (2024), Yang et al. (2026): the authors list these as related work on cross-modal hallucination surveys and detection, without experimental comparison with CaRe.

Mitigation approaches

• Training-time alignment: includes OmniDPO, Mod-DPO, RLHF-V, audio contrastive preference optimization, etc. (Chen et al., 2026; Chaubey et al., 2026; Yu et al., 2024; Baid et al., 2026; Pan et al., 2026; Jiang et al., 2024).
• Representation intervention: includes inference-time intervention, ICT, contrastive activation addition, activation steering decoding, visual information steering, SHARP, and dynamic multimodal activation steering (Li et al., 2023; Chen et al., 2025; Rimsky et al., 2024; Su et al., 2025; Li et al., 2025; Wu et al., 2025; Yin et al., 2026).
• Training-free decoding: includes VCD, AVCD, MAD, visual information grounding, OPERA, and multimodal chain-of-thought hallucination mitigation (Leng et al., 2024; Jung et al., 2026; Chung et al., 2026; Favero et al., 2024; Huang et al., 2024; Ma et al., 2026). The authors state that these methods mainly suppress hallucinated outputs or recalibrate modality contributions.

Baseline methods are the unmodified backbone, VCD, ICT, AVCD, and MAD. Benchmarks/datasets are CMM, AVHBench, and Daily-Omni (Zhou et al., 2025) for general QA.

The authors place CaRe in the training-free, original audio-visual path-preserving setting: using counterfactual divergence to detect recoverable modality misrouting and selectively correcting the joint representation without altering the underlying OLLM.

The authors use CaRe to handle modality misrouting at inference time: a text probe selects a candidate unimodal branch, and only when it hard-conflicts with the joint prediction, the fused representation is corrected along a counterfactual residual, with both modalities remaining on their original paths.

Input configurations and branch decisions

• Three inputs: joint is audio, video, and question; the audio branch withholds visual; the video branch withholds audio. The same frozen OLLM yields B^AV, B^A, B^V. Withholding uses empty inputs; the paper does not mention injecting noise.
• Decision: the candidate set for binary choice is {Yes, No}; for multiple choice, it is the options given in the question. Branch prediction takes the candidate with the maximum logit; the decision margin m is the highest score minus the second-highest, with m=0 indicating no unique preference.
• Scores: z is the next-token logit of the candidate answer label under the evaluation prompt.

Text modality probe

• Input: both audio and video are withheld, estimating the required sensory modality based only on the question. The authors state this design is inspired by MAD's modality self-assessment, but the probe does not look at the current sample's audio or video content.
• Selection: softmax over logits for audio, video, both. Only when the probability of audio or video is strictly greater than the other two is the corresponding counterfactual branch selected; if it points to both or there is no unique maximum, no reference is selected.
• Boundary: the authors state the probe is not assumed to find the factually correct modality, and it does not trigger intervention by itself. Appendix D gives four phrasings: Default and N1–N3; the main experiments use Default, with option order audio, video, both.

Counterfactual conflict gate

• After selecting a reference, it is checked whether it conflicts with the joint prediction. The gate is g = I[r ∈ {A,V} ∧ m_r > 0 ∧ m_AV > 0 ∧ ŷ_r ≠ ŷ_AV]. This formula is 1 only when both sides have unique and mutually different decisions; the authors call this hard counterfactual conflict.
• Samples with g=0 retain the original joint output. When g=1, the selected branch serves as the reference for the residual intervention in Section 5.2.

Adaptive residual intervention

• Residual: on the post-residual hidden state of the final prompt token at decoder layer l, the selected branch is subtracted from the current audio-video state. The current state already includes corrections from earlier layers, so the residual is relative to the evolving representation, not the original unmodified state.
• Strength: the counterfactual margin is mapped via sigmoid to (0,1). Below threshold η, a weak coefficient interval is used; at or above η, a strong interval. The authors' rationale is that predictions barely separating the top candidate have weaker correction signals than decisive predictions.
• Depth decay: over a predetermined K layers (K≥2), the coefficient decays from the upper bound of the first intervention layer to the lower bound of the last layer, with γ>0 controlling the decay. The authors state this allows earlier corrections to propagate backward, while deeper layers make only lighter adjustments.
• Update: the correction is h̃^AV_lj = h^AV_lj + α_j(h^r_lj − h^AV_lj). It only moves the original audio-video state; subsequent computation retains both modalities.

Implementation location

• Only the final prompt token is modified during the prefill phase. The authors state this follows prior activation steering work: the last input token is a compact location controlling subsequent generation. Other prompt positions remain unchanged.
• After prefill, generation proceeds autoregressively from that audio-video state, with no further intervention on new tokens.
• The 7B and MiniCPM-o-2.6 models each have 28 layers, intervening on 25–27; the 3B model has 36 layers, intervening on 33–35. In Table 7, γ is 0.5 for all; η and the weak/strong intervals vary by backbone and dataset. The full procedure is in Appendix C.1.

Experimental setup

• Models: Qwen2.5-Omni-7B, MiniCPM-o-2.6, Qwen2.5-Omni-3B. The first two each have 28 language decoder layers; the 3B has 36 layers.
• Hallucination sets: CMM's Visual Dominance and Audio Dominance each have 200 instances, 400 Yes–No questions, totaling 800 questions. AVHBench uses Video-driven Audio Hallucination (2,290 questions, 1,145 each for Yes/No) and Audio-driven Video Hallucination (1,136 questions, 568 each).
• General QA: Daily-Omni full set of 684 videos, 1,197 multiple-choice questions across six categories; 647 questions for 30 seconds, 550 for 60 seconds.
• Baselines: Unmodified backbone, plus VCD, ICT, AVCD, MAD. VCD is modified to compare against sub-branches of the withheld modality rather than injecting noise (Appendix C.3). ICT, AVCD, and MAD use official implementations integrated into the omni-modal backbone.
• Metrics and judgment: Acc. and F1, with Yes as the positive class. Overall Acc. is computed by merging two subsets or tasks within the same benchmark. Scores come from next-token logits of candidate answer labels, compared against ground truth. The paper does not report LLM evaluation, human agreement, or number of repetitions.
• Protocol: Greedy decoding, temperature 0. CaRe by default intervenes on the last three layers, γ=0.5; η and coefficient ranges are in Table 7.

Main results

Swipe horizontally to read the table

Model	Method	CMM Overall Acc.	AVHBench Overall Acc.
Qwen2.5-Omni-7B	Baseline	70.5	69.8
Qwen2.5-Omni-7B	+MAD	72.5	81.0
Qwen2.5-Omni-7B	+CaRe	85.9	82.3
MiniCPM-o-2.6	Baseline	73.0	73.4
MiniCPM-o-2.6	+MAD	77.3	77.8
MiniCPM-o-2.6	+CaRe	86.9	80.2
Qwen2.5-Omni-3B	Base	76.9	76.4
Qwen2.5-Omni-3B	+CaRe	82.5	81.0

From Table 1 (first two models) and Table 2 (3B). The table lists only the unmodified backbone, MAD, and CaRe for each Overall Acc.; other baselines for the 3B are not reported.

• The authors state that CaRe achieves the highest Overall Acc. on both backbones and both benchmarks, and the highest score on each reported subset. In Table 1, for Qwen2.5-Omni-7B's Audio-driven Video Hallucination F1, CaRe and MAD both achieve 84.5. The authors write that CaRe exceeds MAD by 9.8 and 1.5 percentage points; the corresponding values in the table are 85.9 vs. 72.5 and 82.3 vs. 81.0. MiniCPM's 9.6 percentage points matches 86.9 vs. 77.3.
• On CMM Audio Dominance, the authors state that relative to the strongest baseline, scores improve from 75.5% to 89.8% (Qwen2.5-Omni-7B) and from 72.8% to 88.5% (MiniCPM-o-2.6). In Table 1, the strongest baseline in these two cells is MAD, and CaRe's F1 is 89.8 and 88.3.
• In Table 1, some baselines are lower than the unmodified backbone: for Qwen2.5-Omni-7B, AVCD's CMM Overall Acc. is 69.1 vs. Baseline 70.5; VCD's Audio Dominance Acc. is 66.5 vs. Baseline 71.8. For MiniCPM-o-2.6, VCD and ICT's CMM Overall Acc. are 66.6 and 67.5 vs. Baseline 73.0.

Residual and direct branch selection on gated positive samples

• Tests on g=1 samples compare directly outputting counterfactual predictions (CF-SELECT) versus residual steering. Non-gated samples are identical for both, so they are not included in Table 3.
• On Qwen2.5-Omni-7B, CaRe's Acc. gains over CF-SELECT are Visual Dom. 1.55, Audio Dom. 2.08, V→A 8.84, A→V 9.28 percentage points; corresponding CaRe values are 81.69%, 88.54%, 63.05%, 62.89%. MiniCPM-o-2.6's gains are 6.35, 2.06, 10.82, 3.77, with CaRe at 73.02%, 86.49%, 59.38%, 64.15%.
• The authors state that the benefit comes from treating counterfactual signals as continuous directions and preserving cross-modal context, rather than switching to outputting a single-modal branch. The paper does not explain why AVHBench gated subset Acc. is lower than CMM.

Daily-Omni

• Full Acc. improves from 61.5% to 61.8% (Qwen2.5-Omni-7B) and from 61.0% to 61.5% (MiniCPM-o-2.6) (Table 4).
• The authors state that the most consistent gains across categories are in inference and reasoning: both backbones show 1.3 percentage points in inference (75.3% to 76.6%, 76.0% to 77.3%), and 1.2 and 1.1 percentage points in reasoning. Acc. is also slightly higher on 30-second and 60-second videos.
• Not every category improves: Qwen2.5-Omni-7B's Context Understanding drops from 57.5% to 56.5%, and Comparative from 66.4% to 64.9%. MiniCPM-o-2.6's AV Event Align. stays at 48.7%, and Context Understanding stays at 60.6%. The authors state that overall accuracy is not dragged down.

Routing probes and intervention layers

• Figure 7 includes 96 configurations of prompts × option orders × backbones × benchmarks. The authors state that 85 of these probes have at least 95% agreement with the task-defined target modality; the main experiment's default configuration achieves 99.5%–100% across four settings. In the figure, Qwen2.5-Omni-7B×CMM's N2 row has 50.8, 50.1, 50.6, and the same model×AVHBench's Default row has 89.4, 74.1. The authors state that the signal is largely stable but shows model-specific sensitivity to prompts and option order.
• Figure 4 compares three consecutive layers on CMM. The authors state that early and middle layers yield limited gains; layers 25–27 have Acc. 1.6 (Qwen2.5-Omni-7B) and 0.6 percentage points (MiniCPM-o-2.6) higher than layers 19–21, and 3.8 and 2.5 percentage points higher than random layer selection.
• Figure 5 compares strengths on MiniCPM-o-2.6's CMM. The authors state that adaptive settings have Acc./F1 0.9/0.8 percentage points higher than all-weak settings, and 1.0 percentage points higher than all-strong settings on both metrics.

Other ablations and analyses

• Conflict gating (Table 5, CMM, 800 questions): Qwen coverage 167/800 (20.9%) vs. route-then-intervene 773/800 (96.6%), both with Acc. 85.9; additional interventions 606, Help/Harm 0/0. MiniCPM: 174/800 (21.8%) vs. 772/800 (96.5%), both Acc. 86.9, additional 598, 0/0.
• λα (Figure 8, CMM): Acc. rises from 0.25 to 0.50, then is relatively stable from 0.50 to 1.25; the authors state that the default λα=1 is 0.2 and 0.1 percentage points lower than the best settings. Increasing the scale increases both beneficial and harmful prediction changes.
• η (Figure 9, CMM): Qwen at η=0.1 is 86.1%, default 0.2 is 85.9%, and at larger thresholds the text states it is nearly unchanged at 85.6%. MiniCPM goes from 86.0% at η=0.1 to 86.9% at default 0.3, and at η=0.9 the authors state it is approximately 85.4%.
• Joint λα and η (Figure 10, CMM): Qwen at λα=0.5, η=0.7 has 82.9% in that cell, with the text stating 82.875%; at λα=1.25, all three threshold cells are 86.1%, with the text stating 86.125%. MiniCPM's highest cell is 86.9% (λα=1.0, η=0.3), with the text stating 86.875%.
• Depth schedule (Figure 6, MiniCPM-o-2.6, CMM): decay gives Acc. 0.869, F1 0.873; uniform gives 0.862/0.868; increasing with depth gives 0.864/0.870.
• Latency (Appendix F, 200 examples per backbone, serial): Qwen means CaRe 6.698s, MAD 16.612s, AVCD 4.838s; MiniCPM 4.571s, 7.816s, 2.966s. The authors state that CaRe reduces latency by 59.7% and 41.5% relative to MAD. Ideal parallel estimates are CaRe 4.782s and 3.239s, not measured.

Limitations and future directions noted by the authors

• CaRe currently assumes that useful task-relevant evidence remains accessible from unimodal branches (Limitations and Future Work).
• The conflict gate is defined on a discrete, task-given answer space where branch predictions can be directly compared (same section).
• The authors state that the method may be harder to apply when relevant evidence is weakly encoded, when the wrong path does not manifest as an explicit prediction conflict, or when the task lacks an easily scorable candidate answer space (same section).
• The text probe and counterfactual branches introduce additional inference overhead (same section).
• The authors propose future directions: softer conflict signals, more general routing criteria for open-ended generation, and more efficient single-forward implementations (same section).

Experimental coverage

• Tested models: Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, MiniCPM-o-2.6 (Section 6.1, Table 2).
• Hallucination questions: 800 Yes–No questions from the two CMM Dominance subsets, and AVHBench's Video-driven Audio Hallucination 2,290 questions and Audio-driven Video Hallucination 1,136 questions (Appendix E).
• Baselines for 7B and MiniCPM-o-2.6: unmodified backbone, VCD, ICT, AVCD, MAD; for 3B, classification results only report Base and CaRe (Table 1, Table 6).
• Metrics: Acc. and F1 relative to ground truth, with Yes as the positive class, greedy decoding with temperature 0; the paper does not report number of repetitions, LLM evaluation, or human agreement (Section 6.1).
• Additionally, the full Daily-Omni set of 1,197 questions is used; latency is measured on 200 examples per backbone, and the parallel numbers are ideal branch-parallel estimates (Appendix E.3, Appendix F).

CaRe is a training-free inference-time correction method for audio-visual hallucinations in omni-modal models.

• Problem: Competing modalities interfere with the evidence truly needed by the question. The authors find that many erroneous joint predictions can still be answered correctly by the relevant unimodal branch, and call this gap modality misrouting. Figure 3 gives recovery rate ranges of 52.9%–74.4% on CMM and 37.3%–46.2% on AVHBench.
• Approach: A question-only probe selects the audio or video counterfactual branch; only when both have unique and contradictory answers does it add residuals to the final prompt tokens in the last three layers, scaled by that branch's confidence. The original path retains both modalities, and parameters are not updated.
• Hallucination sets: In Table 1, CaRe's CMM Overall Acc. is 85.9% (Qwen2.5-Omni-7B) and 86.9% (MiniCPM-o-2.6), and AVHBench is 82.3% and 80.2%. On Audio Dominance, CaRe achieves 89.8% and 88.5%. In the same table, Audio-driven Video Hallucination F1 for Qwen is 84.5, tied with MAD.
• Scale and general QA: On Qwen2.5-Omni-3B, CaRe's Overall Acc. relative to Base is CMM 82.5% (+5.6) and AVHBench 81.0% (+4.6) (Table 2). Daily-Omni full set improves from 61.5% to 61.8% and 61.0% to 61.5%; Qwen's Context Understanding drops from 57.5% to 56.5%, and Comparative from 66.4% to 64.9% (Table 4).
• Authors' conclusion: The authors argue that when some audio-visual hallucinations occur, task-relevant evidence is still recoverable from unimodal perspectives; confidence-adaptive, depth-decayed residuals can make targeted corrections in middle-to-late layers. They also state that the method depends on the evidence remaining accessible, and that conflict comparison requires a discrete answer space.