Benchmarks exist so that engineers can test their projects, observe competitors, and compare their performance. Each benchmark serves a specific purpose.

ARC-AGI-3 was created by the ARC Prize Foundation, founded by François Chollet, to evaluate general intelligence and the learning capabilities of AI agents. The premise behind its complex, game-like interactive tasks was that they could only be solved by an artificial intelligence capable of exploring an unfamiliar environment, grasping rules on the fly, planning actions, and adapting to new conditions. In other words, an AI that truly knows how to learn.

A number of projects claim a 100% success rate on this benchmark. But this is not a victory. In this article, I will explain why.

What is ARC-AGI-3

The ARC benchmark was launched in 2019 featuring colored grid tasks, where a model had to solve a problem based on just two examples. Chollet designed it so that each task was governed by a unique rule not found in the training data. This meant the model had to infer it directly on the spot. Chollet himself described this as measuring the efficiency of skill acquisition, rather than measuring the skill itself.

ARC-AGI-1 lasted for five years. Early solutions achieved only single-digit percentages. Later, brute-force algorithms were applied: researchers would create a set of elementary grid operations and run a trivial search for a combination that fit the examples. This worked to some extent, but due to combinatorial explosion, it was impossible to scale. The first real breakthroughs were achieved by reasoning LLMs, which scored over 70%.

ARC-AGI-2 was released as a response in 2025; solving its tasks required combining multiple rules and maintaining context. Current LLMs failed at this.

ARC-AGI-3, the latest version to date, completely changed the task format. The tasks were turned into interactive Atari-style games. Whereas in static tasks agents competed primarily on brute-force computational power, in the games neither the rules nor the goals are known in advance, and they can only be discovered through direct interaction. Moreover, the evaluation accounts for the number of actions taken.

The RHAE metric focuses on two indicators: the number of levels completed and the number of actions it took to complete them. The baseline is an average human playing the game for the first time. The assumption was that if an agent genuinely infers rules from observations, it will require very few actions. If it relies on brute-force, the number of actions will be significantly higher.

How Neural Networks Solve the Benchmark

In essence, three approaches can be identified for neural networks:

Direct interaction. The game provides the model with game states, and the neural network outputs the next action. There is no scaffolding, at most reasoning within the context. The performance is mediocre: the model loses its train of thought, repeats mistakes, and hallucinates.
Reinforcement learning. A classic approach for neural networks, but the benchmark's very design makes it impossible to gather the required volume of training data.
Agentic scaffolding. The model operates in a loop with instructions, memory, a file system, and executable code. This approach has yielded the best results.

I would like to point out that the benchmark, which is supposed to measure learning ability, is actually testing the quality of the engineering scaffolding built around the neural network.

Sergey Rodionov's Project

Let's try to break down the project by Sergey Rodionov, a PhD from SingularityNET, which is a pure example of the third approach. I reviewed his first preprint, then listened to Sergey's presentation with fresh results, and later looked at his second paper, where the author analyzes his own project.

Very roughly, the project's architecture is as follows: the agent is Codex with access to a Linux environment, files, and Python. An algorithmic controller sends prompts to the agent and manages the game loop. The agent observes the state, builds a hypothesis about how the world works, takes an action, checks the result, and refines the hypothesis.

Three ideas were presented in the first preprint:

Executable world model. The benchmark's game task is simulated as accurately as possible in a Python program.
Bias towards simplicity. The agent periodically attempted to replace specific cases with general rules. In other words, a practical surrogate for the minimum description length principle.
Verification by reproduction. The model is required to exactly reproduce every recorded observation. If it doesn't match, the hypothesis is discarded.

My preliminary conclusions after reading the first preprint: the agent complies with the rules of ARC-3, but not with its spirit. This is precisely a coherent engineering construct, where the project is tailored specifically for the benchmark. Such a setup will only work in simple, constrained artificial worlds because it hits two barriers of the exact same combinatorial explosion: writing code for a slightly more complex task, and searching for a solution within the simulation.

But most importantly, tasks continue to be solved by brute force—generating ideas, verifying them, and starting the loop over again. Why does this look like an achievement for the benchmark? Because in this case, hypothesis verification is moved outside the benchmark; it is performed by the LLM-Python simulator system. I raised this point during the presentation, to which I received a completely fair response: who is to say that a human doesn't also solve it through trial and error? I have to agree here. If we set aside the romanticized notions of epiphanies, insights, and visions, by and large, neither science (which is sad) nor philosophy (which was expected) can say anything definitive on this matter.

Ablation

During the presentation itself, Sergey Rodionov managed to surprise me quite a bit. It turns out he had a second paper investigating how each of the three ideas actually impacts the result. In it, he systematically performed an ablation study (turning off one element of his system at a time) and evaluated how this ultimately affected the benchmark's outcome.

First conclusion: Every configuration improves with a more capable LLM and a greater volume of reasoning. Other differences between configurations are smaller than expected, and the effects of individual components are not distinctly pronounced.

In reality, the idea of an executable world model failed to prove its usefulness. The text-based approach (LLM context) outperforms the executable model approach in both configurations of the flagship model. During the presentation, Rodionov confirmed that a separate executable model is unnecessary; the context alone is sufficient for the coding agent.

The Python simulator, which was highlighted in the title of the first paper, lost to simple text-based reasoning.

Full verification ranks first across all four configurations, even though it consumes significantly more resources. Simplification is beneficial in three out of the four configurations.

In other words, a dedicated world model turned out to be unnecessary for the LLM. What is actually useful is idea generation (LLM), their selection, and the compression of descriptions (LLM).

And the result on the best model is roughly 99% RHAE. This leads me to two unpleasant conclusions:

The benchmark does not actually test for AGI, because if it did, ChatGPT would already be that AGI.
Researchers' work on this benchmark using modern LLMs is scientifically pointless; it amounts to nothing more than engineering stress-tests of flagship models.

However, Sergey Rodionov, of course, does not share this view. As I gathered from his talk, the benchmark organizers refused to officially verify his project—presumably relying, as I did, on the vague concept of the "spirit of AGI." This, quite justifiably, frustrates many engineers. If a benchmark has objective metrics that a model meets, then the model should be allowed to participate in the benchmark. If not, then either change the rules or shut the benchmark down.

Unfortunately, it seems that for now, the authors of ARC-AGI-3 are ready to do neither.

Why LLMs Win

Sergey Rodionov's conclusion is that coding agents can serve as universal solvers for deterministic, effectively low-dimensional environments. This is a fairly precise and cautious conclusion. If this holds true, it implies that the frontier of AGI must shift toward operating under uncertainty and the ability to reduce high-dimensional worlds into viable internal models.

Why are LLMs so successful in low-dimensional worlds? I see two reasons: the first is quite logical, while the second is more fundamental.

The first reason: The ARC-AGI-3 games are developed by a small team. In contrast, LLM training data contains decades of work by tens of thousands of game designers: what you shouldn't touch in a game, what you must run away from, what you need to collect, how a level is typically laid out, where to look for the exit. Thousands of highly diverse patterns.

Essentially, the model translates what is happening on the screen into text, searches for familiar gaming patterns, formulates hypotheses, and tests them. The combinatorial explosion is mitigated not by search, but by the fact that the hypothesis space has already been pre-shaped from the outside.

It is highly doubtful that the creators of ARC-AGI-3 have more imagination than the professionals who design games for a living.

In other words, the nature of this success is not on-the-fly rule inference, but the recognition of learned concepts. An LLM acts as an interactive library: it does not inherently solve the problem; rather, it finds a similar pattern in its database and tries it out. It is still a brute-force approach, but a smart brute-force approach. This, admittedly, is an achievement in its own right. A library of this scale and accessibility has never existed before.

The second reason: Recall Kahneman. System 1 is fast thinking, System 2 is slow thinking. By definition, System 2 cannot process the sheer volume of data that System 1 digests. In fact, if we were to feed System 2 that same volume of data, its very existence would lose all meaning. This is exactly why Sergey's Python world model offered no benefit. System 2 must be able to operate on abstractions, but no one can clearly define how to properly abstract the external world within a deterministic environment.

An LLM, however, does this effortlessly by virtue of its design. It describes the world through text, through language. A pixel that destroys the player-controlled pixel upon contact is simply labeled a "hunter." This immediately triggers a stream of hypotheses from the LLM based on learned patterns: run away, hide, find a weapon... This is not thinking in the human sense, but in low-dimensional worlds crafted by humans, it is more than enough.

In short, for an LLM, textual description is the native form of abstraction required for System 2. And this is exactly why it performs better than a perfectly precise Python simulation of the game.

What the Benchmark Cannot Measure

This brings the structural problem of ARC-AGI-3 into plain sight.

RHAE measures an objective result: how many levels are passed and how many actions it took. The benchmark has no access to the method used to achieve that result. Brute-force and genuine thinking are indistinguishable when both produce the same answer on the benchmark's tasks.

Action cost was supposed to act as a control mechanism—the idea being that brute-forcing is expensive, while understanding is cheap. However, penalties are only applied to actions taken within the game system, whereas actions within the simulation are not counted. An agent can endlessly run hypotheses in Python or through text-based reasoning, and then simply submit the correct solution. In other words, size, speed, and raw power win out over understanding.

This is not a loophole; it is a direct consequence of the fact that the benchmark measures the model's output rather than the internal mechanics of the solution. Then again, no one is able to offer an alternative approach anyway.

The Main Problem

Let me try to explain what I mean by genuine thinking. Let's take a simple task and a small neural network.

A micro-network is trained to determine whether a sequence of numbers is sorted—say, sequences of length 3 to 10. It learns perfectly, achieving an accuracy of 1.0, but if fed a sequence of length 14–20, it fails. Neither inductive biases, nor introducing a second micro-network with a different length range, nor quadrupling the network's size helps.

However, if we introduce a delta (the difference between adjacent numbers), the micro-network successfully solves the task and generalizes the solution to new lengths.

In other words, a solution exists, it is simple, and it is contained within the data, yet the network fails to see it.

And this makes perfect logical sense. In the network's world—that is, within the distribution of the training data—the delta is always the worse option: it is more computationally expensive and more costly to represent, all while yielding the same result. The network optimizes strictly relative to this specific distribution. The requirement to "work on any length" is a demand imposed solely by me.

I conducted a series of experiments: training on a continuous range of lengths, using a probe to reconstruct the delta from hidden states, and measuring attention on a single-position shift. The result: the task is successfully solved within the baseline sequence lengths, but beyond that boundary, accuracy drops, and the delta never emerges in the network's internal representations.

The network simply finds a surrogate solution that works only on the training lengths.

I tried expanding the distribution, applying pressure to minimize the representation via an information bottleneck, introducing a second network with a different length range while demanding representational agreement, and so on. All these variations only worked within the training domain.

In short, the neural network is incapable of altering its variable space (i.e., introducing the delta), even though doing so would simplify the solution (compress its description) to an absolute minimum—the delta simply doesn't change its sign. (Here, the criterion becomes local: if the sign of the delta remains constant across the entire sequence, it is sorted. The check is no longer dependent on the sequence length; each position is evaluated independently, and the results are merely aggregated). The only way around this is to explicitly feed the delta from the outside directly as data, or to modify the architecture so rigidly that nothing other than the delta could possibly emerge.

The Reason

Backprop optimizes a function. It fundamentally does not alter the variable space: what changes are merely the coordinates within the parameterization pre-established by the architecture—via the number of layers, the attention mechanism, and what a specific position can or cannot see.

Loss compares behavior against a criterion. But when searching for a new representation, that criterion is absent. Loss does not and cannot dictate which variables must be introduced to solve the task.

To achieve the goal, the representation space itself must be made a self-modifying object of learning. But how do you evaluate the alteration of the space itself if the objective function is already fixed? Backprop has no answer for this. And there is a strong suspicion that any internal network add-ons operating via backprop will be equally unhelpful.

If someone manages to design a neural network in such a way that it inherently derives the delta (or its analog) for the sequence sorting task without it being directly or indirectly fed into the system, I will gladly throw this entire section out.

But for now, it comes down to this:

An LLM is the search for a solution within a predefined representation space.
AGI is the search for a representation space in which a solution becomes achievable.

And intelligence is the ability to find a data representation in which the target expectation becomes achievable.

What Might Work

If the answer is not inside the system, the remaining option is to look outside.

One could try training on a drifting distribution rather than a fixed one. The conditions of the task change faster than a solution can solidify for the current ones. A surrogate breaks with every shift; a discovered invariant does not break.

Meaning, the criterion comes from the external world.

However, I am confident that something else—something that isn’t backprop—will also be required.

Conclusion

In its spirit, ARC-AGI-3 was conceived as a benchmark of understanding, not automated brute-force. In reality, it currently measures how well a coding agent, armed with the experience of others, handles a deterministic, low-dimensional world.

Naturally, this is not a grievance against the benchmark or against those trying to solve it. Sergey Rodionov did excellent engineering work and did not hide its weak spots.

The grievance is with the very idea of measuring intelligence by the result. The benchmark cannot distinguish between a rule and a surrogate; the difference lies deeper, where the benchmark naturally cannot reach. Furthermore, the probability of inventing a task that is not based on patterns contained within the global dataset of human knowledge is exceptionally low. And LLMs are in a winning position here.

LLMs are effective interactive libraries: they know how to find what fits a new task from accumulated knowledge, and they are doing it better and better. But they do not know how to search for a space in which a solution becomes simple.

And this is possible, otherwise mathematics as a science would not exist. Complex numbers, planetary orbits, set theory—all of these are consequences of humans altering their representation space.

P.S. Perhaps the search for a new space could be orchestrated through an LLM. Language, as a coherent construct, can describe anything, including things that were never in the data, up to and including hallucinations. But whether this would just yield another book from the library, I cannot yet say.

AGI Benchmark. If ARC-AGI-3 is solved, do we have AGI?