A team of researchers posted a preprint on arXiv on October 16 introducing DialFork, a benchmark to measure procedural hallucination in conversational AI agents.
While factual accuracy of large language models has been widely studied, the paper argues that another failure mode—procedural hallucination—has been overlooked. Procedurally hallucinated dialogues are individually fluent and plausible but deviate from the intended conversational flow. DialFork is described as the first benchmark to capture this error type on raw dialogue text, according to the preprint.
The dataset contains 36,278 dialogues built by forking conformant conversations at controlled turn positions and injecting one of four hallucination types. It spans 24 tasks across 12 domains, with construction validated by 30 human annotators. From this data the authors define three evaluation tasks: a set-level task for monitoring deployed agents, a single-dialogue procedural retrieval task, and a turn-level task for real-time guardrails that identifies when a dialogue first goes off-track.
To measure procedural deviation, the paper introduces two dialog-flow-aware metrics, FlowKL and FlowScore. Unlike standard semantic similarity measures, these compare how dialogues unfold. Across seven encoders, FlowScore significantly outperformed all baselines at the dialogue level, demonstrating, the authors write, that “semantic similarity is no substitute for procedural similarity.”
The work is a preprint and has not yet been peer-reviewed. The authors release DialFork under an MIT license, with the hope that the community will “measure conversational process alongside content.” No independent replication of the benchmark or the proposed metrics is yet available.