Forty little black-and-white puzzles just threw a bucket of cold water on the autonomous-agent sales pitch. Einsia AI’s Navers Lab says its new Visual Cipher Bench found that the best result among eight frontier models was only 26.7%, with no tested configuration clearing 30%. That is not a minor blemish. It is a reminder that a model can sound clever, use tools, and still stumble badly when nobody hands it the playbook.
The benchmark, announced November 18, 2026, is deliberately unfriendly to the usual AI comfort zone. There are 40 original puzzles across 67 illustrations, and agents get one broad instruction: inspect the images and write a final answer to a file. No explicit rules. No helpful question spelling out the task. No immediate signal saying whether a step was right. In other words, closer to the messy ambiguity people mean when they talk about genuine autonomy.
The Missing Skill Is Not Just Visual Recognition
Across 3,480 attempts and 29 model-effort configurations, the results draw an awkward line between executing a known rule and discovering one from scratch. GPT-6 Astra at its maximum setting led at 26.7%, or an average 10.7 correct answers in a 40-puzzle run. GPT-6.1 Sol and Claude Opus 5.5 each managed 21.7% in their best tested settings.
Then the floor falls out. GPT-6 Luna and Claude Sonnet 5.5 solved one puzzle apiece across 120 attempts, scoring 0.8%. DeepSeek V4.1 Flash, Qwen 3.8 Max, and Kimi K3.1 Pro produced zero correct answers. That spread is worth noticing, but the larger message is harsher: even the leader was wrong nearly three-quarters of the time.
Spatial operations were the relative bright spot, at 22.8% accuracy. The other categories were rough reading: knowledge decoding hit 3.3%, logical planning 2.4%, and rule induction just 0.6%. Five correct answers out of 870 rule-induction attempts is not the sort of number anybody should bury beneath a glossy demo reel.
Hints Change the Game, Which Is the Point
The most revealing experiment may be the simplest one. On 10 puzzles, Einsia gave GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna the core rules without supplying the intermediate reasoning or final answers. Overall accuracy jumped from 12.2% to 51.1%. Rule-induction performance leapt from 0% to 85.2%.
That is a massive improvement, and it tells us something useful rather than merely embarrassing. These systems were often capable of applying a rule once it was surfaced. They were lousy at reliably finding and validating that rule on their own. GPT-6 Astra rose from 23.3% to 70.0%; GPT-6.1 Sol climbed from 13.3% to 66.7%. Timeouts across 90 attempts dropped from 51 to 29.
Autonomy is not just getting an answer after the instructions arrive; it is figuring out what the instructions are when the world refuses to label them.
A control test adds another wrinkle. Extra information about task-state representation, without the core rules, delivered only 16.7% accuracy. Logical planning also remained weak at 22.2% even when hints were available. So this is not a one-bug story where a better prompt fixes everything. Rule discovery and multi-step planning are separate headaches, and both are still very much alive.
What Sector Watchers Should Take From This
For anyone watching AI deployment rather than admiring benchmark leaderboards, this is the part that matters. The pressure to market agents as independent workers is intense. Businesses want systems that can inspect unfamiliar situations, choose a path, and finish a job without a human babysitting every edge case. Yet the benchmark’s failure patterns sound painfully familiar: agents formed premature conclusions from incomplete evidence, or burned time on brute-force tool use instead of narrowing possibilities from visual constraints.
That does not make the technology useless. It makes the deployment boundary clearer. Structured workflows with explicit requirements, strong test feedback, and narrow operating ranges remain far more defensible than vague, open-ended tasks. The source notes say some correct answers came from thematic guesses rather than systematic rule discovery, which is exactly the kind of result that can look fine in a small pilot and become expensive in production.
Benchmarks Need More Than Victory Laps
Visual Cipher Bench is only 40 puzzles, so nobody should pretend it settles the entire question of machine reasoning. Small benchmarks can be sensitive to design choices, and this release reports results from its own research group. Still, the gap is too wide to shrug off: no model exceeded 13.3% in knowledge decoding, logical planning, or rule induction, even while the top model reached 80% in spatial operations.
That contrast is the real signal. Progress is uneven, not magical. Companies building or buying agents should ask where rules come from, how a system tests its own assumptions, what happens when feedback is absent, and when a human must step in. Until those answers get sturdier, the grand claims about hands-off AI operators deserve a raised eyebrow, not a blank check.