A preprint on arXiv proposes a fixed rule for deciding when an automated pipeline may act on a language model's answer, and a record that lets each decision be replayed.

Pipelines that let a large language model (LLM) make discrete decisions, such as whether an alert is benign, need to know which outputs they can act on unattended and why each was accepted, the authors write. The paper, "Anchor and Check," is not peer reviewed, and its pages report no independent replication.

According to the paper's introduction2, sampling methods such as self-consistency, SelfCheckGPT and semantic entropy cost 5 to 10 generations per item and reduce the evidence to a score. The authors ask how much of their error-detection power a smaller procedure can recover if the extra calls go to samples that make different mistakes.

The procedure commits the target model's greedy answer, called the anchor. It then draws one or two more greedy answers, either from an alternate prompt or from a model of another family. A pure function compares them and returns one of five outcomes, and only "Agree" is acted on automatically.

The authors say the checks themselves are not new, citing SAC3 and work by Xue and colleagues, and that their contribution is a deterministic contract around them plus careful measurement. They also write that they do not claim anchor preservation or replayability is unique to the contract, since any baseline can store its samples for replay. What the contract adds, they say, is a committed answer that never changes with sampling, an explicit five-way outcome instead of a score threshold, and a decision replayable from its record.

In the authors' tests, covering 48,000 model-item evaluations over 6,000 questions1, every method had access to verbalized confidence as a tie-breaker. As complete pipelines, the three-call contract and pooled semantic entropy over 3 or 5 samples differed by less than one point, with intervals containing zero.

As detectors of the anchor's errors, the contract was ahead of pooled entropy with 3 and 5 samples under both tie-breaker conventions. The introduction puts the lead at 3.9 points over 3 samples and 1.5 over 5 under one convention. With 10 samples the point estimates differ by less than one point, and the sign depends on the convention.

The authors say most of the detector lead comes from where the answers are drawn, not from the scoring rule: entropy over the same three greedy answers scores only 0.6 points lower. Pooling another family's samples raised semantic entropy's AUROC by 1.7 to 4.0 points at every budget, and a same-model temperature check was the weakest design.

This report draws on the arXiv abstract and the introduction of the HTML version. The full PDF was unavailable, so the models, tasks and limitations described elsewhere in the paper could not be checked.

Under the strict "Agree" gate, the contract automates 70% of items and still acts on 16% of errors, according to the abstract. The authors add that generation itself was not reproducible on the hosted API they used.