Researchers at the Allen Institute for AI released StateBench, a benchmark measuring how language models track state changes across long documents, in a paper posted to arXiv2 and a Hugging Face blog post1 on November 3, 2026.
StateBench gives researchers a finer-grained way to test where state-tracking breaks down. Existing tests have centered on the hardest class of problems; builders now need to know which simpler forms each architecture can handle as recurrent and hybrid designs become more common, the authors wrote.
The benchmark arrives as a growing body of work maps the theoretical limits of different architectures on state tracking. Transformers have well-documented weaknesses on the hardest variants with long dependency chains, motivating designs such as Gated DeltaNet that carry a running memory forward. An earlier preprint, “The Illusion of State in State-Space Models,” found that models such as Mamba and S4 face similar limits despite their recurrent formulation. Gated DeltaNet and hybrid architectures have since emerged, raising the question of precisely which levels of state tracking each can solve.
StateBench sorts state-tracking problems into four levels drawn from Krohn–Rhodes theory, a branch of algebra. The easiest, r-trivial, involves immutable assignment. Aperiodic tasks allow reassignment by value. Soluble problems add self-referential updates. The hardest, general finite, permits assignment by reference—one variable copying another’s current value. The authors derived the three easier levels from the same general finite source trace so the four levels differ only in assignment type, allowing fair comparison.
In the paper the authors argue that “the class of problems transformers are thought to length-generalize on does not even contain all aperiodic state tracking” and prove that Gated DeltaNet with positive eigenvalues expresses all aperiodic problems. The paper is a preprint and has not undergone peer review; the benchmark and all results are the authors’ own evaluation.
Frontier models falter on harder levels
The authors tested four frontier models—Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna—on StateBench, asking each to read a sequence of variable assignments and generate the final value of every variable. All four performed well on r-trivial and aperiodic problems, but accuracy fell sharply on soluble and general finite problems as inputs grew longer, dropping to 13% or lower on the longest general finite problems, according to the AI2 blog post. The arXiv abstract states that none of the pretrained models robustly solves even soluble state tracking.
The blog post does not report prompting configurations or per-model breakdowns, and the retrieved paper page does not supply them. Several open pretrained models proved even weaker: Olmo 3, a 7-billion-parameter transformer, and Olmo Hybrid, which adds Gated DeltaNet layers, stayed near chance on general finite problems. Results also varied by problem format—a 275-million-parameter hybrid and Gated DeltaNet scored near the ceiling on RULER-style r-trivial problems but were near chance on the same problems framed as spreadsheet edits.
Direct training closes the gap for some
To separate architectural limits from what pretraining teaches, the researchers trained several 275-million-parameter models from scratch on StateBench sequences of up to 256 assignments and tested them on sequences as long as 64,000 assignments. Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic, and soluble problems at the longest lengths tested. Transformers learned the easier levels but degraded on longer sequences and never learned general finite problems.
On the hardest level the Gated DeltaNet variant with a more flexible memory update generalized farthest, staying near perfect out to roughly ten times its training length, the blog post states. The authors wrote that for Gated DeltaNet and hybrid models the gap points to training rather than architecture; trained directly, they handled levels their pretrained counterparts could not.
Open questions
The authors’ new proof, described in the blog post, shows that Gated DeltaNet with a more limited memory update can solve all aperiodic tasks and certain periodic ones under fixed-precision arithmetic with rounding. Its exact placement in the hierarchy remains open, including whether it can solve all soluble tasks and how numerical precision or combining it with attention layers affects its capabilities. StateBench also assumes a fixed number of variables, while real tasks often introduce new entities over time, a limitation the current framework does not address.