Does a Higher Score Mean a Better Robot?

Foundation models now enter robot systems in the four ways of Figure 1, and each way puts the model in charge of different parts of the system. Most benchmarks score all four with one success rate over a collection of test cases, and that number cannot say which part the model got right.

Where the agent enters

We formulate a task as a POMDP¹⁴

• \(\mathcal{S}\) is the state space of the robot and objects.
• \(\mathcal{A}\) is the set of controller targets.
• \(\mathcal{O}\) is the set of camera and proprioceptive observations.
• \(T(s' \mid s, a)\) is the transition model.
• \(Z(o \mid s', a)\) is the observation model.
• \(G \subseteq \mathcal{S}\) is the absorbing goal set.
• \(b_0\) is the initial belief over the scene.

The robot knows \(\mathcal{A}\) and \(\mathcal{O}\). The rest of the notation is

• \(\theta = (G, T, Z)\), the unknown task, with \(\theta^*\) the true one.
• \(\ell\), the goal evidence (an instruction or a demonstration), with likelihood \(p(\ell \mid G)\).
• \(h_t = (\ell, a_{1:t}, o_{1:t})\), the history the policy acts on.

The objective is

Since \(\theta^*\) is unknown, \(\pi^\star\) acts on the belief \(b_t(\theta) \propto p(\theta)\, p(h_t \mid \theta)\) and gathers information when that raises success, as in a Bayes-adaptive POMDP¹⁵.

The four designs of Figure 1 differ in how much of \(\mathcal{M}\) the foundation model \(f\) must specify itself. Two more symbols are needed.

• \(\Pi\) is a set of skills \(\pi_k(a \mid h; \kappa)\), controllers with parameters \(\kappa\) that run until they terminate, as options in a semi-Markov decision process¹⁶.
• \(h^{(e)}\) is the history of attempt \(e\).

Agent as prior. The agent must understand the whole problem and implement it. Before acting, \(f\) writes a model \(\hat{\mathcal{M}} = \langle \hat{\mathcal{S}}, \hat{\mathcal{A}}, \hat{\mathcal{O}}, \hat T, \hat Z, \hat G, \hat b_0 \rangle\), or a policy that implies one, complete enough for a program, planner, behavior tree, or MPC to act with no further help. This is the highest requirement of the four designs, and \(f\) sees the effect of its choices only between attempts, through \(h^{(1:e-1)}\). Many systems write only part of \(\hat{\mathcal{M}}\) and take the rest from hand-written code. RIR, for example, writes only the goal \(\hat G\)⁴. A vision-language model proposes candidate goals for a demonstration, a planner scores how well each goal explains the demonstrated motion, and RIR keeps refining the best candidates. This is the belief update above made explicit, with the model's proposals as the prior \(p(\theta)\), the planner's score as the likelihood \(p(h \mid \theta)\), and the kept candidates as the posterior.

Hierarchical VLA. Decoupling lowers the requirement. A learned policy \(\pi_{\mathrm{VLA}}(a \mid h_t, g)\) takes over \(T\) and \(Z\) at the level of motion, so \(f\) only chooses the next subgoal \(g\) and planning reduces to reasoning over subgoals. The decoupling is lossy, because choosing well requires the policy's ability boundary, the probability \(\rho(g \mid h_t)\) that \(\pi_{\mathrm{VLA}}\) reaches \(g\) from the current history, and nothing in the design shows \(\rho\) to \(f\). SayCan has to learn a value function for each skill to estimate it⁸.

Skills as tools. A tighter connection between the two levels addresses this. Each skill call returns its outcome to \(f\), so its history \(\bar h_t\) records where each skill in \(\Pi\) succeeds and fails, and \(f\) can create, modify, or redescribe skills to match. Keeping what it learns pays off. ASPIRE writes, repairs, and stores skills across tasks, and its library reaches 31% success on unseen long-horizon LIBERO-Pro tasks, against 4% for earlier methods¹¹.

Agentic control. As \(f\) learns what its skills can do, their abilities merge into \(f\) itself, until \(\Pi\) shrinks to the low-level controller and \(f\) sends actions in \(\mathcal{A}\) directly. How far this works today depends on the interface. Claude Plays Robotics found that a model's score "depends as much on the robot body and the control interface as on the model itself", and with direct arm control on LIBERO tasks its models completed 0% to 5.5%¹⁷. With VIA's interface for setting gripper targets, the strongest agents reached 85% and 100% on six simulated tasks and 24 of 25 real episodes¹³. On RoboARCade, though, VIA adds only 1.5 to 6.8 points over raw multi-view RGB-D for the two strongest agents, at 1.7 to 2.3 times the cost¹⁸.

Benchmarks

Table 1 lists the test classes of five benchmarks, what the robot receives, and example tasks.

LIBERO, ManiSkill3, and RoboDojo are collections of test cases, and their classes describe tasks rather than abilities. LIBERO's first three suites each vary one factor, the layout, the object, or the goal, but its policies train and test on the same tasks, so a suite tests recall rather than inference²³. ManiSkill3 groups tasks by robot and setting, gives no instruction, and fixes each task's success condition. RoboDojo's classes group very different tasks. Its generalization class includes arranging number tiles into the largest possible number, and its long-horizon class puts tic-tac-toe next to tidying a desk.

KinDER isolates physical reasoning from perception and language. The robot observes the object-centric state, and each environment fixes its goal. Many environments carry two or three of these tags, ScoopPour3D all of tool use, nonprehensile manipulation, and dynamic constraints, so a score on one class does not measure one ability. The coding-agent follow-up changes what is handed over, not what is tested. Its agents receive a task description and the simulator instead of KinDER's skills, predicates, operators, and samplers, so they must build these themselves³.

RoboARCade is built for agentic control (Figure 1D). The agent is the policy and sends every end-effector command itself, through either raw multi-view RGB-D or VIA's interface, so the two interfaces can be compared on the same tasks. Its 22 long-horizon tasks run on three robots. Most adapt scenes from earlier benchmarks such as LIBERO, OGBench, and RoboDojo with wider initial-state randomization, and its classes name abilities without tagging each task with one. The agent sees no object poses or success predicates, a rubric grades each subtask from the simulator state, and every failed subtask is labeled by the first step that went wrong, visual understanding, approach, or precision, along with any false claim of success¹⁸.

What the benchmarks show

Figure 2 shows which benchmarks are close to saturated.

Table 2 lists what these results show.

Read together, these results say where current methods break. Coding agents that build their own abstractions now beat every other method, and planning over hand-written abstractions slows and fails as objects multiply. No method handles contact-rich dynamics. The planner's hand-written abstraction leaves them out, models learned from demonstrations predict them poorly, and the coding agents' programs fail on the same dynamic 3D tasks. Agents that act directly rarely misread the scene. They choose a poor approach or execute it imprecisely, and they often declare success too early. Of the components below, transition and constraint extraction is the one current methods get wrong most consistently, and the one a benchmark most needs to isolate.

The benchmark we need

A physical agent cannot improve itself by trial and error on the real robot, because real trials are too slow and too hard to reset for the many attempts self-improvement takes. Explicit reasoning therefore puts a simulator between the agent and the world (Figure 3). The agent first extracts the simulator from the real world, then builds its policy and skills from the simulator's feedback, and at inference chooses which of them to run on the robot, reading each real outcome as it goes. The simulator is any model that returns feedback for a proposed action. It can be a physics engine such as Isaac Sim, a planning domain, a program, or a markdown harness that describes the scene and its rules in text. Reasoning can also stay implicit inside \(f\), as in a VLA, but recursive self-improvement (RSI) needs feedback on each proposal, and only an explicit simulator gives it.

A benchmark can score the three parts separately. It scores abstraction by comparing the simulator an agent extracts with the true task, reasoning by giving the agent the true simulator and testing the skills it builds, and inference by giving it fixed skills and testing which ones it chooses. Together the parts cover the five things the formulation asks an agent to infer. State extraction, goal specification, and transition and constraint extraction belong to abstraction, skill grounding to both reasoning and inference, and action reasoning to inference.

• State extraction. \(\hat s_t = \hat\phi(h_t)\), the task-relevant state recovered from the history, which inverts \(Z\) and decides which properties in \(\mathcal{S}\) to keep.
• Goal specification. \(\hat G\), the goal inferred from \(\ell\).
• Transition and constraint extraction. \(\hat T\), how the scene responds to actions, including kinematic, geometric, and dynamic constraints.
• Skill grounding. \(\hat\rho(g \mid h_t)\), the agent's estimate of the ability boundary of the policies and skills it builds with the simulator, so that the subgoals and skill calls it chooses are ones the robot can carry out. Asking the agent to predict the outcome of each call and comparing the prediction with what happens measures it.
• Action reasoning. Ordering actions by their dependencies under \(\hat T\). A later action may need the result of an earlier one, as a bowl can go into a drawer only after the drawer is opened, and an earlier result may block a later action, as filling the front of a shelf first leaves no way to reach the back.

No benchmark in Table 1 scores these components separately. Doing so takes three things, isolating each component, probing it, and making sure the score depends on it.

1. Isolate one component. Give the robot the true value of every other component, so that a failure can only come from the one under test. To test goal specification, for example, give the robot the true state and a simulator and ask it only for the goal. KinDER does this for planning, since its planners all receive the true state, goal, skills, and models. Entries must also report everything they were given, because extra access can leak a component. Coding agents that could read KinDER's source code could read the goals in it³.
2. Probe it with paired scenes. Make two scenes that differ in one property, and check that the robot's behavior changes exactly when the true task says it should. On one KinDER sorting task, an agent's program put cubes into bins by the order in which they appear in the state, not by their color. With four cubes this happened to work, and with twenty it failed every time³. A pair of scenes with the same order and different colors would have caught the error at once. Contrast sets and HANS test language models the same way^{26, 27}. When the agent writes a component as code, such as RAPID's success tests, compare the code with the truth directly².
3. Make sure the score needs the component. A robot can pass a task without the component in two ways, by recall or by accident. Drawing test tasks from a generator that no one has seen rules out recall²⁸. Running policies that do nothing or act at random measures accident. On KinDER's BalanceBeam3D, the goal checked only that the beam was level, which it already was at the start, so doing nothing passed until the goal was fixed²⁹.

From score to the real robot

A simulated score says something about real robots only after its correlation with real-robot success has been measured. That takes a small set of real tasks, each with a simulated copy of the same scene. Evaluate the same systems on both, publish the rank correlation between simulated and real success with a confidence interval, and recompute it for every version of the benchmark. Such copies are getting cheaper. KinDER rebuilt a real shelf scene from an overhead camera²², and RAPID reconstructs a scene from a single RGB-D image².

Published correlations are low or uncertain. RoboDojo reports only partial agreement between its simulated and real rankings, without a coefficient²¹. SIMPLER ranked six policies with a Pearson correlation of 0.924³⁰, but with six policies the 95% confidence interval runs from about 0.45 to 0.99 (my calculation). The PolaRiS authors found LIBERO poorly correlated with real performance³¹. Prediction-powered inference can correct simulated estimates with a few paired real trials³².

A benchmark without a measured real-world correlation is a hypothesis about robots, not a measurement of them.

Open questions

• Should a benchmark fix the interface? Fixing the interface fixes what \(f\) outputs. Comparing designs needs a benchmark that lets each entry choose its interface and reports the choice.
• Who writes the hidden tasks? A generator of unseen tasks needs goals that a program can check, and such goals are easy to get wrong. Besides BalanceBeam3D, three KinDER environments had goals that were missing, unsatisfiable, or checked the wrong region, and they were corrected in September 2026²⁹.
• How many paired real tasks are enough? Real trials are slow, so the paired tasks may have to be shared across labs, as RoboArena shares real-robot evaluation³³.

        @misc{higher-score-better-robot-blog,
          author    = {Hu Tianrun},
          title     = {Does a Higher Score Mean a Better Robot?},
          year      = {2026},
          publisher = {GitHub},
          url       = {https://h-tr.github.io/blog/posts/higher-score-better-robot.html}
        }