Orivael · ARC-AGI-3 · Public results
ARC-AGI-3 is the interactive reasoning benchmark from ARC Prize. Unlike earlier ARC tasks these are games: a 64×64 grid of coloured cells, a handful of buttons, no instructions, and no explanation of what any of it means. You work out the rules by playing. ft09 and tr87 are now finished 6 of 6, at 100%, with no language model in the loop.
Download the full write-up PDF · 7 pagesft09
6 of 6 levels · 80 actions
tr87
6 of 6 levels · 322 actions
Inference spend
No LLM in the loop, anywhere
RL floor
Episodes solving bp35 level 1
Every result below was returned by ARC Prize's own scorecard API when the run was closed. Each carries a replay.
| Game | Levels | Actions | Official score |
|---|---|---|---|
| ft09 | 6 of 6 | 80 | 100.0% |
| tr87 | 6 of 6 | 322 | 100.0% |
| cd82 | 2 of 6 | 21 | 8.59% |
| bp35 | 2 of 9 | 93 | 6.67% |
| lf52 | 2 of 10 | 42 | 5.45% |
Relative Human Action Efficiency — your actions against a first-time human baseline, squared, weighted so later levels count for more.
Taking twice as many actions as a human doesn't score half. It scores a quarter. A single level caps at 115; the game total caps at human.
On ft09 the human baseline is 208 actions across six levels; we finish in 80, and every one of the six hits the 115 cap. Several other levels cap too: cd82 level 1 in 7 actions against a human's 55, lf52 level 1 in 8 against 32.
The scorecard's own per-level figures. 322 actions, no resets, against a human total of 414.
| Level | Our actions | Human baseline | Level score |
|---|---|---|---|
| 1 | 28 | 54 | 1.150 capped |
| 2 | 30 | 58 | 1.150 capped |
| 3 | 39 | 40 | 1.052 |
| 4 | 29 | 45 | 1.150 capped |
| 5 | 50 | 71 | 1.150 capped |
| 6 | 146 | 146 | 1.000 |
Levels 5 and 6 are the ones nobody had opened. Level 5 inverts the puzzle: both words are fixed and the legend itself is what you edit — you are handed a translation and asked which rules produce it. Dialling it out isn't an option; eight panels of seven glyphs is 5,764,801 states. What makes it finite is that the rules have to tile both words exactly, which leaves 24 orderings, and most of those ask a panel for a glyph it can't display.
Level 6 is every earlier mechanic at once — one glyph rewriting as several, longest-first segmentation, an editable legend, rotated glyphs, and a chain through a colour that appears in neither word. The last level of one of these games seems to be an exam on the rest of it.
No LLM is in the loop — not for perception, not for planning, not for choosing a move. The agent reads the raw grid, decides, and acts. Total inference spend across every run above: $0.00.
That isn't a cost story, it's a capability one. Under a squared efficiency metric, an agent that narrates a board in natural language and then guesses is competing against a human who can simply see the answer. The ceiling is where the points are, and you don't reach it by being verbose. It also makes the runs deterministic: same agent, same game, same 80 actions, every time.
Almost every failure was a confident, plausible reading of the board that was wrong about the board — and none of them raised an error.
The clearest example cost a full day on bp35. Every probe read the board at reset and treated it as fixed. On that reading the level is unwinnable: you click every visible block and the exit never opens.
Visible blocks go from 147 to 273 on the fourth step right — identically in our own build and in ARC's recording. Two of the blocks the solution needs are simply not on screen when the level starts.
The reason we didn't see it is worth more than the fix. It had been raised early — the
whole level can't be seen — and was argued down because the camera reports x=0, y=0
and never changes. That's true, and irrelevant. The camera object doesn't move; the
content does. A real signal was outweighed by a precise measurement of the wrong thing.
Once probes re-scan after movement, bp35 level 1 is 15 actions against a human baseline of 21 — the score cap. (Verified by replaying the sequence in our own build; the bp35 card above predates it.)
Others in the same family, all real:
Exhaustive over what was sampled, reported as exhaustive over what exists.
A separate class, and worse in a different way: the measurement apparatus corrupting the thing it measures, while continuing to emit plausible numbers.
Row 63 of the bp35 board is an action counter. It gains a cell every action. Our reasoner hashed the raw grid to decide whether it had seen a state before — so every state was novel by construction. One run reported 40 distinct states in 40 actions. One hundred percent new ground is not a good result; it's impossible. Revisit detection, the untried-action frontier, and the retrospective's are-we-going-in-circles check were all reading a counter.
We misidentified the player four times — three times as that same action bar, once as a static decoration. Conservation isn't enough, because when the player moves the floor's cell count is conserved too. The player is the blob that translates: same shape, new position. Seeding from the smallest blob picked a five-cell ornament sharing the avatar's colours; taking the largest instead moved one game from 1 reachable position to 36.
ACTION7 is undo. It was in the datasheet ARC provides. Our harness had it
filed as a wasted action to avoid, so a free probe — try something, look, take it back — sat
unused while we spent real actions on the same question.
A UTF-8 byte-order mark blanked the first key in a .env file. PowerShell's
Out-File -Encoding utf8 writes one, so the parser saw an invisible U+FEFF
glued to the front of the key name and matched nothing. Only the first key in a file is ever
affected, which is exactly the kind of bug that survives a careful read.
Each was nearly read as a result.
$0.00 spent,
byte-identical results. A paid model that costs nothing did not run.In none of these was the tell the headline metric — the headline metric looked fine. The tell
was $0.00, or identical output across repeated runs of a stochastic model.
Refuse to print an A/B comparison when fallback usage exceeds a threshold. Print VOID instead. Any agent evaluation with a fallback path can otherwise report a clean number for a component that never ran.
Before claiming any agent result, we measured the floor on bp35 level 1.
| Policy | Episodes | Solved |
|---|---|---|
| Random — moves + clicks | 200 | 0 |
| Random clicks only | 200 | 0 (never died) |
| Random moves only | 200 | 0 (died 66.5%) |
| Tabular Q-learning | 1,500 | 0 |
So the level is a real test rather than a lottery, and any result has to be reported against zero rather than against nothing.
Q-learning is the instructive one. Its death rate fell from 59.0% to 28.5% — it genuinely learned something. It learned to survive, never to solve. The death signal is learnable from raw grid hashes; the solve signal isn't.
tr87 was a replay and is not any more. For a long time we opened four of its levels by reproducing a recorded human play-through, which demonstrates nothing and cannot reach a level nobody recorded. The score above is derived: the agent reads the board's own legend, works out the rules, and dials the answer. Levels 5 and 6 are ones no recording could have given us.
Most of that gain was our own waste, not new reasoning. Finishing the game moved tr87 from 25.99% to a projected 67%. Removing redundant measurement moved it from 67% to the cap without changing a single deduction — the agent had been reading every dial's full cycle before turning it, on every slot of every level, when the first match was provably the only one. It would be easy to present the whole jump as insight. Most of it was us deleting our own overhead.
We over-reported that run before the card corrected us. A level can score 115 and four of tr87's do, so our own scorer computed a game total of 109.3%. The API caps the game at human and returned 100.0. The scorer was wrong, is fixed, and the number here is the card's.
Twenty of the twenty-five public games are untouched, and one we've looked at we cannot take a single legal action in — its controls share nothing with the games we've solved. Averaged over the full public set rather than the five we've played, these numbers are small.
Level counts vary and later levels are worth more. Two levels of a nine-level game is 6.7%, not 22%. Where we're stopped, we're usually stopped by a mechanic that arrives without warning: a block that comes back when you click it twice, a platform that ferries a piece between boards, a legend that is itself the thing you have to solve.
ARC-AGI-3's difficulty isn't precision. Where we can identify a game's mechanic, we tend to beat the human baseline outright. It's recognising the game at all — every one of these games teaches its own rules through its own first level, and an agent that arrives carrying the last game's assumptions gets nothing.
Can a system recognise what kind of world it has entered, without importing assumptions from the last one?