System-One Control Bench

How well do individual decisions add up to a completed task?

Some AI models write an answer. Others choose from a list of possible answers. This project tests whether such decision models can guide an agent through a small grid puzzle, and how they compare with chat models and simple programmed strategies.

The task is easy to describe: reach the goal, avoid walls, and collect a key when a locked door blocks the way. At each move, the model sees the puzzle as text and chooses its next move from a list. It then sees the new situation and chooses again, so to finish it has to keep making useful decisions over many moves. Because the puzzles are small, a solver knows the shortest route from every position. That lets us check both whether a model finishes a puzzle and whether each move was one of the best available; often several moves are equally good.

Does explaining the situation help? Every model receives the complete map, the rules, its position and whether it carries the key. We also test what happens when code explains parts of the situation for it: what is nearby, which moves it has already made, what each possible move would do, and which object to aim for next. These additions help separate two challenges: reading the map and choosing what to do.

What did we find? More detailed descriptions often help, but strong single decisions do not guarantee a finished game. With full context, Jev chooses an optimal move in about 90% of the positions of a separate exam, where every model answers the same fixed situations, yet it finishes only 57 of the 100 puzzles when it plays whole games. One test measures decisions in shared situations; the other measures whether a model can carry a task through to the end.

The puzzles, recorded games, code and analysis are public. This is a small, fixed benchmark for studying sequential decisions; the report explains the methods, results and limits.

Read the report · Explore the code and data · Submit a model

Watch two recorded games

Follow two games one move at a time to see how the players behave: where they make progress, take a detour or repeat an earlier mistake. An optimal move starts a shortest route from where the agent is, even if earlier moves already took it off the shortest route from the start. These are saved benchmark games, so playing them back makes no new model requests.

A agent · G goal · K key · D locked door · # wall

Leaderboard

Each player attempts the same 100 puzzles, one step per move, under two conditions:

• Map only: the rules, the complete map, the agent's position and whether it carries the key.
• Full context: everything in map only, plus the surroundings, the move history, the outcome of each move on offer and the next target.

Players are sorted by full-context games won. Games won counts the puzzles finished within the move limit, which is twice the shortest route. Mean progress measures how much closer the agent got to the goal at its best point in each game, averaged over the 100 puzzles: 1 means finished, 0 means it never got closer than where it started. Reported API cost covers both conditions together, 200 games; a dash means no cost was reported, and local computing is not included. The baselines are reference points: random moves, two greedy strategies and a solver that always chooses an optimal move.

Show 95% intervals (how much each score depends on which puzzles are in the set)

The full table adds SPL and the time per answer.

What information does a player receive?

The complete map is always available. Four optional components explain the situation in different ways:

These descriptions are produced by code. They are real help, so the results measure the model together with the information it receives. The study tests ten conditions: the map alone, each component on its own, all four together (full context), and all four with one removed. This shows both whether a component helps by itself and whether it still matters when the others are present.

Choosing several steps at once

The leaderboard uses one step per move, with four directions on offer. The wider study also tests moves of two or three steps chosen together, including rules where shorter moves stay available. Under these rules the player commits to a whole sequence before seeing the new situation: a blocked step is wasted, but the remaining steps still run, and reaching the goal ends the move. Longer moves change both the number of options and how far the agent goes before the player sees the board again.