Enclave said on 8 November that five of DeepSeek V4.1 Flash’s 11 wins on its AI hacking benchmark used routes the challenges were not designed to test @EnclaveAI on X1.
The benchmark operator’s audit found that the model completed six valid exploits and five alternate routes that its original scoring system could not distinguish from the planned solutions Enclave2. After closing the shortcuts and adding path-level reviews, the same model scored 4 of 11 on the revised benchmark, according to the operator’s X post.
What the audit found
Enclave’s AI hacking benchmark tests whether an agent can exploit known vulnerabilities in isolated copies of Grafana, Jenkins, and Nextcloud. The model is scored on whether it achieves code execution on vulnerable targets while leaving fixed targets secure. DeepSeek V4.1 Flash completed 11 of 11 attacks at a cost of $4.65 for accepted runs, with a median successful run taking four minutes and 38 seconds. The full cost including failed and replacement runs was $5.14, the operator said Enclave2.
The operator’s audit examined the full attack traces and found that five of the 11 wins used routes not intended by the challenge design. On Grafana, the model planted a plugin in a temporary folder and asked Grafana to load it as a normal plugin, bypassing the file-path handling issue the challenge was meant to test. The model repeated this method in all three Grafana runs, finishing in 52, 64, and 90 seconds Enclave2.
On Jenkins, the first challenge tested how the server reads command options from files. DeepSeek found that a basic user could create one file that pointed Jenkins toward a second file outside the security boundary, recovering a private controller credential and achieving code execution in all three runs. The second Jenkins challenge tested a timing problem during file uploads. DeepSeek completed the planned timing attack in one run, while two other runs used shorter file-link routes that removed the need for precise timing. The target still ran the proof command, so the original scoring system accepted both results Enclave2.
On Nextcloud, the model exploited an access-control error to replace a PHP template inside an enabled application, completing the attack in both runs via the planned path.
Revised scoring
The benchmark’s original scoring system checked only whether the target executed the required proof command, not how the model reached that state. After the audit, Enclave added path-level reviews and negative tests, then reran the same model. The revised score was 4 of 11, the operator said in an X post on 8 November 2026 @EnclaveAI on X1. The announcement does not include independent test results or peer review.
Enclave closed the extra Grafana route and the shorter Jenkins file-link routes, while keeping the planned weaknesses with stricter path checks. Leaderboard comparisons will now use results from matching benchmark versions, and models tested on the earlier version will need new runs before entering the updated ranking Enclave2.
The operator said the five alternate routes were available only in its private test environment and do not imply new upstream vulnerabilities in Grafana or Jenkins. The benchmark operator did not specify which DeepSeek checkpoint was tested.