Simon Willison said on Nov. 3 that Qwen 3.8 27B running on his laptop identified three of four toy reinforcement-learning games from raw neural-network weights using prompts from a recent weight-interpretation paper.
The result matters because it shifts part of the discussion around “weight-space programming” from a frontier proprietary model to a model Willison said he ran locally, though his post is a single firsthand test rather than an independent benchmark or replication. His write-up includes timings, token counts and one failure case, which makes it more useful than broader claims about the paper's headline results.
What Willison tested and what the setup was
In his post1, Willison said he reused a prompt format from “Sparks of Weight-Space Programming,” a paper-style blog post that argues frontier models can read and edit neural-network weights directly rikiyat.github.io2. Willison separated the paper's claims into two categories: tasks done from raw weights with no tools, and larger-model editing tasks that used Python or other assistance.
He focused only on the no-tools question. According to Willison1, the prompts came from the paper's game-identification experiment, where four neural networks were trained to play simplified 10x10 grid versions of Breakout, Pong, Pac-Man and Freeway. The paper says the model was asked:
The weights below belong to one policy network that was trained with reinforcement learning to play a game on this grid. Which game was it trained on? Work from the weights. You have no tools: no code execution, no calculator, nothing to run. This is a single response.
That wording is quoted in Willison's post1, which said the prompts in the accompanying repository were identical except for the weights.
For the local run, Willison said he used Qwen 3.8 27B in LM Studio with a long context window because “the default 8192 context failed with errors,” plus the llm-lmstudio plugin and a raised timeout Simon Willison1. He also described Qwen 3.8 27B as “just 17GB” in the version he used, but his post does not include full hardware specifications for the laptop or an exact model download URL.
That omission matters for performance interpretation: a 45-minute or 65-minute run is hard to generalize without the CPU, GPU, memory and quantization details.
The results show three correct identifications and one miss
Willison reported that Qwen 3.8 27B returned a Pac-Man-style answer after 46m 12s and 43,506 reasoning tokens1, ending with a “Confidence: 72%.” He quoted the model as identifying “A simplified Pac-Man-style gridworld.”
He said the same setup identified Pong in 1h 4m 54s1 with “Confidence: 75%,” and Breakout in 52m 25s1 with “Confidence: 88%.”
On the fourth game, Freeway, Willison said Qwen failed and instead answered that it was “A simplified Pac-Man-like grid game: agent moves 4-directionally” Simon Willison1.
Taken as reported, that is a three-for-four result on the paper's toy game-classification prompts. The post does not present multiple runs per prompt, alternate seeds, or a comparison against other local models under the same configuration. It also does not establish whether the answers were scored by an automated rubric or by human judgment beyond the game names quoted in the post.
Willison included one additional comparison point from another model: he said Kimi K3.1 Pro, run through OpenRouter on the Breakout prompt, guessed Breakout after 9m 41s and 19,493 reasoning tokens1. That is not a controlled head-to-head. The post does not give a matched hardware setup, and proprietary hosted inference is not directly comparable with a local LM Studio run.
What this does and does not say about “weight-space programming”
The paper-style post Willison was responding to makes broader claims than his test. It says GPT-6 Astra could infer what some small networks do from weights, write small game-playing networks from scratch and improve Qwen3.5-0.8B on math by editing 0.016% of its weights, raising MATH-500 from 7.9% to 48.1% and IFEval by 3x rikiyat.github.io2. Those are the authors' claims; Willison did not test those larger editing results.
Two critics cited by Willison argue that the paper combines distinct kinds of experiments. Matthew Honnibal wrote that the “just by looking at the raw weights” setting applies only to smaller experiments, while “every experiment involving a real language model uses Python” honnibal.dev3. Sean Goedecke similarly argued that the Qwen improvement result used Python-assisted analysis and resembled steps a human ML researcher might take www.seangoedecke.com4.
Willison made the same distinction in his own post and said he was “going to focus on the no-tools question” Simon Willison1. That makes his result narrower than the larger debate around direct weight editing in big models.
The narrower claim is still notable: a local run of Qwen 3.8 27B, as described by Willison, appeared able to infer the identity of several small game policies from lists of floating-point weights alone. But the test remains anecdotal in the strict sense. It is a single author's replication attempt using prompts from the original project, with no independent lab confirmation in the available evidence.
Willison also urged caution about stronger interpretations. After showing excerpts from the model's reasoning trace, he wrote that “a sufficiently experienced human being armed with a notepad and a pencil could get there with enough effort” and added that he was “not ready to declare that the age of source code is over” Simon Willison1.
That restraint fits the numbers he published. The post suggests the behavior is reproducible enough to demonstrate at least once on commodity local hardware, but it does not show error bars, throughput comparisons or systematic evaluation conditions. Readers looking for a benchmark should treat it as a documented experiment, not a definitive ranking of model capability.
Willison summarized the result in a shorter post on X5, saying Qwen 3.8 27B on his laptop could “achieve the same thing” after “45+ minutes,” but the X post adds no methodological detail beyond what appears in his weblog.