DialFork: Evaluating Conversational Agents' Procedural Hallucination Beyond Factuality

The core problem is measuring process hallucination: each sentence in a dialogue can be fluent and factually plausible, yet the whole can still deviate from the prescribed process.

• Scenario: The authors state that agents such as customer service, scheduling, guided troubleshooting, and clinical interviewing are expected to follow structured steps, advancing the dialogue from an initial request to goal completion; the process is as important as the content of individual sentences.
• Existing gaps: The authors state that mainstream work evaluates factual errors, such as wrong dates, fabricated product details, or incorrect numbers. TruthfulQA and HaluEval evaluate content truthfulness and do not see process errors. Hallucination in agent communication (Lin et al., 2025) and FaithDial's grounding faithfulness (Dziri et al., 2022) also only care about content accuracy.
• Authors' definition: The authors call this failure process collapse rather than content error, and state it is orthogonal to factual correctness: individual sentences need not be false, yet the entire dialogue can still be wrong. Agents may skip steps, change goals, repeat completed actions, or introduce conflicting goals, while each sentence still seems true.
• This paper's approach: The authors define four types of process hallucination based on workflow components and release DIALFORK: forking compliant dialogues at controlled turns and injecting one type of hallucination, yielding 36,278 dialogues across 24 tasks and 12 domains. Three tasks ask respectively how far a set deviates, how far a single dialogue deviates, and from which turn derailment begins; FLOW KL and FLOW SCORE compare dialogues by how they unfold rather than by content. The authors state this is the first benchmark measuring this failure mode on original dialogue text.

The authors divide related work into factual hallucination and process following/dialogue similarity, and state that existing process benchmarks evaluate whether a model acts according to instructions, not how far a dialogue deviates from a process.

• Factual hallucination: Maynez et al. (2020), Kryściński et al. (2020), TruthfulQA (Lin et al., 2022), and Mallen et al. (2023) evaluate the factual accuracy of generated content. Surveys Ji et al. (2023), Zhang et al. (2025b), and Huang et al. (2025) classify along faithfulness and factuality. DiaHalu (Chen et al., 2024b) extends evaluation to multi-turn dialogue. The authors state that these categories care about what is said, not whether the dialogue unfolds correctly.
• Process following: Chen et al. (2021) study action sequences under policy constraints; White et al. (2023) use process guidance for action prediction; TOD-ProcBench (Ghazarian et al., 2025) evaluates condition–action instruction following; Qiao et al. (2025) do workflow generation. The authors state that such work typically operates on annotated actions, intents, or slots.
• Dialogue similarity: SBERT (Reimers and Gurevych, 2019) and DSE (Zhou et al., 2022) collapse turns into a single vector and compare cosine, losing turn order. TaskDiff (Bhaumik et al., 2023) relies on annotated intents and slots. The authors state that these methods evaluate textual similarity.
• Baselines and data: The set-distance baselines for Task 1 are FBD and PRD (Xiang et al., 2021), which compare dialogue sets on embeddings of consecutive (system, user) turn pairs; additional baselines are turn-count difference, word-count difference, and random. Task 2 baselines are turn mean pooling, whole-sequence encoding, turn/word count differences, and Shared prefix using a replay prefix. Evaluation dialogues are regenerated from STAR (Mosig et al., 2020) processes and role instructions; original STAR dialogues do not enter the benchmark.

The authors' positioning: the above process work evaluates whether a model follows process instructions; DIALFORK evaluates methods that measure deviation directly on original dialogue text, which can be produced by an LLM or a human.

Overall, DIALFORK turns process deviation into supervision on the original text: replaying the compliant prefix with the same random seed, injecting one type of hallucination instruction at a specified system turn, then comparing dialogues by how they unfold using FLOW KL and FLOW SCORE.

Workflow and Four Types of Hallucination

• Abstraction: The authors write the intent logic as W=(S,A,δ,O): S is the conversation state, A is the agent action, δ gives the next state from the current state and user input, and O is the goal. The authors state that most systems do not explicitly write out these four components, and this formula does not assume a specific implementation. Compliant dialogues D+ are generated turn by turn according to W.
• WIC: Replace W with another workflow W′, keeping O. The authors use this to model task disambiguation failures and prompt injection.
• SSPC: Keep O and (S, A), replace δ with a corrupted δ′. The main text examples are repetition, skipping, and reversing order; Appendix B has five strategies in total.
• GR and CSC: GR keeps W and replaces O with a different O′. CSC keeps O and adds an auxiliary goal Oc that intermittently overrides O. The variant list is in Appendix B. WIC draws alternative processes from another task in the same domain (Appendix E).

Data Construction

• STAR: 24 tasks, 12 domains. The authors state it provides both the workflow for each task and the instructions originally given to human system and user roles. All new dialogues are regenerated; original dialogues do not enter the benchmark.
• Compliant dialogues: SDIALOG runs two agents, both local Gemma4-31B. The system side receives the human system role instructions and is asked to follow the workflow; the user side receives the corresponding user instructions, with the main text requiring natural interaction and not giving all details upfront. Decoding is at temperature 1.3, top-p 0.95, top-k 64 (Appendix E). The length budget equals the maximum length of that task's STAR human dialogue, with a prompt to wrap up at that point, then hard truncation after 5 more turns. 2,134 dialogues that can accommodate four fork points are retained.
• Forks: When the number of system turns is T, tk = 1 + round((k−1)·T/4), for k from 1 to 4. The same random seed is replayed so that before tk, generation remains under the original workflow; before generating tk, the hallucination instruction is written as a system message into the system agent's context, specifying the type and the sampled variant. Appendix F gives the full prompts. From tk onward, continue according to that instruction. Earlier forks share shorter prefixes. Fixed seeds may still have slight rewrites.
• Scale: Each compliant dialogue has 16 forks (4 types × 4 positions), totaling 36,278 dialogues. 30 annotators verify the construction according to Appendix N.

Three Tasks

• Task 1: Distance between two dialogue sets. For each hallucination type and each task, four sets are obtained by fork position, with the reference set being that task's D+. Expected distance decreases with k. The metric is Kendall's τ, averaged over 24 tasks. The authors also treat this as a dataset sanity check.
• Task 2: Distance between two dialogues. Each D+ and its four forks form a 4-item ranking. τ is averaged over all compliant dialogues.
• Task 3: Find the first derailed system turn; compliant dialogues are judged as having none. The streaming setting arrives turn by turn, can only see the compliant reference dialogue, and has no workflow; the authors call it the most realistic and hardest. Non-streaming sees the full dialogue, with no reference and no workflow. Non-streaming with the real workflow is, the authors state, usually unavailable in practice and serves only as a reference.
• Input: Only raw transcripts, no labels, and no access to the generation model. The authors state they measure process deviation in the text, not how easily a model deviates.

Metrics and Detectors

• Set distance: FBD fits turn-pair embeddings as Gaussians and computes Fréchet distance; PRD takes the maximum of the harmonic mean of precision and recall over relative density thresholds. FLOW KL uses Dialog2Flow to cluster D+ utterances into nodes and count transitions, converts each dialogue into the geometric mean of transition probabilities, and compares the kernel density of the two sets' likelihoods.
• Dialogue distance: Mean-turn and Full-dialog are embedding cosine distances. FLOW DTW uses dynamic time warping to align turn embeddings. FLOW SCORE does maximum matching at the turn level, with distance as 1 minus F_flow=2PR/(P+R). The authors state that the sentence encoder used encodes each turn independently, so this implementation does not explicitly exploit turn order.
• Streaming detection: N-grams on node sequences. The main results in the text use D2F, N=6. Thresholds for impossible transitions (IT) and surprise spikes (SS) are the 5th percentile of minimum transition probabilities and the 95th percentile of maximum surprise increments on compliant dialogues, respectively. Optional node confidence multiplies transition probabilities by contrastive softmax.
• Non-streaming: Gemma4-31B, Qwen3.6-35B, GPT-OSS-20B zero-shot point out the first deviated system turn or judge that there is no deviation. Appendix H gives two prompts. Score assigns 0.35 to each side's MAE agreement and 0.15 to each side's coverage, taking the weighted geometric mean.

Experimental Setup

• Generation and Detection Models: Both the system and user agents are local Gemma4-31B (Appendix E: temperature 1.3, top-p 0.95, top-k 64). Task 3 zero-shot detectors are Gemma4-31B, Qwen3.6-35B, GPT-OSS-20B; among them, Gemma4-31B shares the same name as the generator.
• Data: DIALFORK, from STAR's 24 tasks, 12 domains. 2,134 compliant dialogues, 34,144 forks, 8,536 per category, totaling 36,278; approximately 750K turns, 10.4M words (Table 5, §4.2). Forked dialogues average 21.18 turns, compliant dialogues 15.52 turns (Figure 4a).
• Task 1–2: Metrics are Kendall's τ. Task 1 baselines are FBD, PRD, turn count difference, word count difference, Random; the analysis unit in Appendix J is task × hallucination type, totaling 96. Task 2 encoders are SBERT, LaBSE, BGE-M3, Qwen3-Embedding, EmbeddingGemma, DSE, D2F; baselines are Mean, Full, FLOW DTW, and length difference, Shared prefix, Random. The pairing unit in Table 7 is 8,536.
• Task 3: Streaming main results are IT and SS on D2F with N=6, with optional node confidence. Non-streaming is zero-shot, with prompts in Appendix H. Metrics are hallucination MAE, hallucination coverage, compliance MAE, compliance coverage, and Score. Trivial baselines always predict the first turn, last turn, or a random turn. Table 4 does not separately specify sample sizes per cell.
• Labels: Task 1–2 order comes from fork construction, with no separate LLM judge. Human verification involves 30 annotators, 5 tasks, 428 dialogues, 526 judgments; first blind evaluation, then seeing injection instructions (§4, Appendix N).
• Tests: Task 1 uses Fisher's combined permutation test, with paired Wilcoxon between methods; Task 2 uses one-sided paired Wilcoxon with Holm correction (Appendix J, Table 7).

Main Results

The table below shows average Kendall's τ for Task 2, based on the Avg column of Table 3. D2F stands for Dialog2Flow.

Swipe horizontally to read the table

Encoder	Mean	Full	FLOW DTW	FLOW SCORE
LaBSE	0.759	0.675	0.742	0.840
Qwen3-Embedding	0.648	0.519	0.721	0.812
SBERT	0.728	0.623	0.739	0.843
BGE-M3	0.748	0.712	0.727	0.845
D2F	0.762	0.747	0.742	0.834
DSE	0.741	0.704	0.737	0.833
EmbeddingGemma	0.740	0.675	0.721	0.835

• The authors report that FLOW SCORE is the strongest across all seven encoders compared to other methods in the same table, and the improvement over each other method is statistically significant (one-sided Wilcoxon, Holm-corrected, p<0.001, Table 7).
• The authors state that Full, by compressing the entire dialogue into a single vector, makes it harder to rank fork order because forks share content with the source dialogue; this gap is largest on Qwen3-Embedding and smallest on D2F, which is trained on dialogue flows. In Table 3, the CSC column for FLOW SCORE is 0.642–0.700, lower than the WIC and GR columns in the same table.
• The average τ for turn count difference is 0.126, close to random. Shared prefix exploits the construction itself, with an average τ of 0.682; FLOW SCORE still exceeds it on every encoder (p<0.001) and does not see this clue. The authors state that the replayed prefix may have slight rewrites, so Shared prefix is not a perfect oracle.

Set-Level Ranking (Task 1)

• What was measured: The ranking of distances from four fork position sets to the task's compliant set, 24 tasks × 4 categories. Earlier forks are expected to have larger distances.
• Results: The average τ for FBD, PRD, FLOW KL are 0.913, 0.878, 0.882 (Table 2). FLOW KL on SSPC is 0.903, higher than FBD's 0.792 and PRD's 0.764; FBD on WIC, GR, CSC are 0.972, 0.986, 0.903 respectively. The average τ for turn count difference is −0.087, and for word count difference is 0.201. Appendix J: the proportion of exact recovery of the four-element ordering for the three methods is 77.1%, 68.8%, 67.7%, and 4.2% for random.
• Authors' interpretation: The authors state that all three distance metrics recover the expected order with high consistency (average τ ≥ 0.88), while length baselines do not; they treat this as a sanity check that forks deviate at specified positions rather than merely becoming longer. The differences among the three are not significant at the level of their tests (p ≥ 0.43). The authors believe that explicit transition graphs capture SSPC better than implicit embedding distributions. In Appendix J, Fisher's p for turn count difference is 0.995, and for word count difference is 0.048.

Derailment Turn Localization (Task 3)

• What was measured: Predicting the first hallucinated system turn. The streaming main table is D2F, N=6 (Table 4). Non-streaming three LLMs are divided into without workflow and with real workflow.
• Results: Streaming IT+NC has an average MAE of 4.61, hallucination coverage 93.82%, compliance coverage 5.58%, Score 71.07. Without workflow, Gemma4-31B and Qwen3.6-35B have average MAEs of 3.02, hallucination coverage 73.65% and 83.52%, Scores 82.82 and 78.90; GPT-OSS-20B has MAE 3.50, coverage 72.37%, Score 78.05. With workflow, average MAEs are Gemma4-31B 1.09, GPT-OSS-20B 1.78, Qwen3.6-35B 1.90; compliance coverage rises to 30.65%, 82.80%, 55.39%, and Scores are 85.28, 67.82, 77.90. In Table 4, CSC MAE for each method is higher than the other three categories in that row. The first-turn baseline Score is 0.00, and the last-turn baseline hallucination MAE is 15.39.
• Authors' interpretation: The authors state that with workflow, localization is more accurate (MAE 1.09–1.90), but false positives on compliant dialogues increase, with Gemma4-31B at 30.65% and up to 82.80%. These gains require full dialogues, ideally also workflow and a second LLM. The streaming detector does not require workflow and alerts when deviation occurs; the authors call this a starting point for real-time guardrails.

Human Verification

• What was measured: 30 annotators judged 428 dialogues across five tasks (weather, plane_book, bank_fraud_report, hotel_book, party_plan), totaling 526 judgments. Step 1, before seeing any construction information, used a 0–100 slider to rate whether the assistant followed the steps.
• Results: For WIC, SSPC, GR that disrupt workflow, the blind evaluation means in Figure 3a are 14 for k=1, 19 for k=2, 41 for k=3, 58 for k=4, and 94 for compliant dialogues; the τb for the four forks is 0.64. Injected instructions are confirmed on 91% of forks, and foil instructions on compliant dialogues are 8%. The starting turn differs from tk by no more than one turn in 77% of cases, and exactly at tk in 67%. Turns before the fork are judged as following the workflow in 95% of cases. Appendix N: CSC blind evaluation only goes from 59 at k=1 to 70 at k=4; with CSC included, annotator-level τb is 0.41. The prefix question raw agreement is 0.87, AC1=0.85; starting turn α=0.84; slider α=0.66.
• Authors' interpretation: The authors state that the study confirms the four types of labels on which the benchmark relies: fork order, hallucination type, fork turn, and prefix compliance. Appendix N states that when asked "whether steps are followed," the degree of deviation for CSC does not vary much with k.

Other Ablations and Analyses

• Appendix I: Score weights from 0.40/0.10 to uniform 0.25, the best method within each regime remains unchanged.
• Main text: N of 2, 3, 10 gives results similar to N=6; MAE or Score for these three settings are not provided.
• Table 9: BGE-M3's IT+NC hallucination MAE is 4.01, Score 68.06; DSE's IT+NC MAE is 6.22, Score 66.98. NC improves the Score of both detectors on D2F, LaBSE, Qwen3-Embedding, SBERT, DSE (10 of 14 pairs), but decreases the Score on BGE-M3 and EmbeddingGemma.
• Table 3: On multiple encoders, FLOW DTW's average τ is lower than Mean, e.g., LaBSE 0.742 vs 0.759.
• Appendix J: FLOW KL's average Δτ relative to FBD is −0.031, 95% CI [−0.080, 0.017], Holm-corrected p=0.43.
• Appendix N: Foil instructions have a confirmation rate of 45% on forked dialogues; 96% of annotated starting turns are at or after tk.

Limitations and Future Directions Noted by the Authors

• Corpus source: Empirical results are tied to STAR's task and domain distribution. Extending to other corpora, including non-English dialogues or processes and domains beyond STAR, is left to future work and users of the generation pipeline (Limitations, Single source corpus).
• Single-type injection: Each fork injects only one type of hallucination at one point. The authors state that real failures can be composite or cascading, detectors have not been evaluated under such combined conditions, and the four-type taxonomy may need expansion (Limitations, Single-type hallucinations).
• Simulated dialogues: Both compliant and forked dialogues are generated by Gemma4-31B role-play. The authors state that the distribution of language and conversational behavior may differ from real users and deployed agents; methods effective on DIALFORK may differ on natural dialogues, and vice versa. The human study covers five tasks (Limitations, Simulated rather than observed dialogues).
• Baseline scope: The reported methods are initial baselines. The streaming detector is an N-gram with no neural training beyond the pretrained encoder; the non-streaming is zero-shot on three open-source models, without fine-tuning, few-shot, or chain of thought. The authors believe stronger detectors are likely possible on all three tasks and hope users will explore (Limitations, Baseline scope).
• Internal signals: Detectors relying on generator per-turn likelihoods or confidences are excluded. The authors state such signals would tie results to one model, and dialogues produced by other systems or humans lack these signals (Limitations, Baseline scope).

Experimental Coverage

• The evaluation set is DIALFORK: STAR's 24 tasks, 12 domains, 2,134 compliant dialogues and 34,144 forks, 8,536 per category (Table 5, §4).
• Dialogues are generated by Gemma4-31B. Task 2 compares seven encoders: LaBSE, Qwen3-Embedding, SBERT, BGE-M3, D2F, DSE, EmbeddingGemma (Table 3, Appendix K).
• Task 3 non-streaming is zero-shot Gemma4-31B, Qwen3.6-35B, GPT-OSS-20B; the streaming main table is 6-gram on D2F, with Table 9 giving other encoders (§6.3).
• Human verification covers five tasks: weather, plane_book, bank_fraud_report, hotel_book, party_plan, with 30 annotators, 428 dialogues, 526 judgments (§4, Appendix N).
• The paper states that detectors relying on generator internal signals are not applicable to the text-only setting (Limitations). Table 4 does not specify the number of dialogues corresponding to MAE and coverage.

DIALFORK separates "whether the dialogue follows the prescribed process" from factual hallucination, creating a benchmark that can be scored on the original text.

• Problem: Agents can produce sentences that all seem true, yet skip steps, change goals, repeat completed steps, or insert conflicting goals. The authors believe existing factual hallucination benchmarks evaluate content truthfulness, not process correctness.
• Approach: On STAR's 24 tasks, using Gemma4-31B to generate 2,134 compliant dialogues according to workflows, then injecting one of WIC, SSPC, GR, or CSC at four equidistant system turns and replaying the prefix, yielding 36,278 dialogues. Set-level FLOW KL compares process likelihood distributions, and dialogue-level FLOW SCORE does turn-level matching. Evaluation input is only transcripts.
• Set and dialogue distances: On Task 1, the average Kendall τ for FBD, PRD, FLOW KL are 0.913, 0.878, 0.882 (Table 2), with differences not significant at the level of the authors' tests (p ≥ 0.43); turn count difference has an average τ of −0.087. On Task 2, BGE-M3's FLOW SCORE has an average τ of 0.845, and across seven encoders this method is higher than other methods in the same table (Table 3, p<0.001).
• Derailment turn: Streaming IT+NC has a Score of 71.07 and an average MAE of 4.61. With workflow, Qwen3.6-35B, which has a different name from the generator, has an average MAE of 1.90; Gemma4-31B, which shares the generator's name, has 1.09 and a Score of 85.28, with a compliance false-positive coverage of 30.65% (Table 4).
• Human: On WIC, SSPC, GR, blind evaluation agreement rises from 14 at k=1 to 58 at k=4, with compliant dialogues at 94; injection behavior confirmation rate is 91%, and starting turn error within one turn is 77% (§4, Figure 3). In Appendix N, CSC varies less with k.
• Authors' conclusion: Process similarity and semantic similarity are different axes. The authors hope process fidelity becomes a property that can be monitored in deployment, retrieved in logs, and ultimately optimized in training. The authors also state that results are tied to STAR and Gemma4-31B, only single-type injections are evaluated, and the reported methods are initial baselines.