What Are We Actually Benchmarking in Robot Manipulation?

TL;DR: A team at TTIC audits the five most-reported robot manipulation benchmarks and finds that the two most popular ones — LIBERO and CALVIN — are largely broken as evidence of capability. A 0.09B-parameter model with no language encoder (it looks up the task by index) matches or beats reported state of the art on three of four LIBERO suites. Only ~20% of published SOTA claims on LIBERO are provably statistically significant from the numbers papers report. And resampling CALVIN's block poses from within the training distribution drops every tested policy's score, some by 25 percentage points on the full-chain metric. The paper packages these findings as four reusable diagnostics — shortcut solvability, statistical significance, creeping overfitting, data-source dependence — with reference implementations, so this is less a takedown than a toolkit the field can now be held to.

The proxy problem

A benchmark score is an average success rate over one fixed test distribution. What we actually want to know is whether a policy has general manipulation capability — the thing that would let a robot tidy a home it hasn't seen. The entire value of a benchmark rests on the score being a valid proxy for that capability, and in March 2026 alone, 79 arXiv papers reported LIBERO numbers. At that rate, LIBERO isn't a testbed; it's the field's operational definition of progress.

The ideal validity test — train everything on shared data, evaluate on an agreed set of real-world tasks, check that rankings match — is impractical. There's no agreed real-world task set, and retraining every policy is too expensive. Checking cross-benchmark ranking consistency doesn't work either, because authors choose which benchmarks to report and tune until they hit SOTA on those, so new papers claim SOTA everywhere they report.

So the authors do what vision and NLP did a decade ago (VQA v2, the PASCAL horse-copyright-tag story, ImageNet-v2, CIFAR near-duplicates): they define concrete failure modes and test for each one directly. All four share a single root cause — the score is an empirical average over a narrow fixed distribution, treated as evidence about a much broader one — but each diagnostic catches a distinct way the two can come apart.

Diagnostic 1: Can a policy without the capability get the score?

LIBERO is a language-conditioned benchmark. A high score is read as evidence of instruction understanding, broad generalization, and the value of large-scale robotics pretraining. The authors' probe has none of these: ~90M parameters, a DINOv2 ViT-B/14 image encoder plus a small MLP head, no diffusion head, no action tokenizer, and — crucially — no language encoder. Because LIBERO draws instructions from a fixed public set, the probe replaces the instruction with a randomly initialized per-task embedding it looks up by index. It's a pure "memorize the task menu" policy, using only what the official protocol allows any submission to use.

The probe beats OpenVLA (7B) by 11–39 points and lands within one point of the best published result on three of four suites. That's the VQA "blind yes-sayer" moment for manipulation: the score is achievable without the capability it's supposed to certify. On CALVIN the shortcut is real but weaker — the probe matches widely used policies (e.g., 3.24 ATC vs. RoboFlamingo's 2.48 on ABC→D) but not the current best (~4.8). The important control: on RoboTwin 2.0 the same probe trails the best by 35+ points, on RoboCasa by 60, and on SimplerEnv it scores 0%. The shortcut is a property of specific benchmarks, not a universal trick.

The authors are careful about the logic, with a nice analogy: if a written driving test can be passed by memorizing the answer key, passing it isn't evidence of driving ability — but people who pass can still be good drivers. High-scoring VLAs may well be capable; the score just no longer proves it. (One honest caveat, disclosed in Appendix A.1: the probe's reported numbers use best-of-several-checkpoints selected on the test suite, since the benchmarks provide no validation set. Final-checkpoint numbers are 0.6–4.4 points lower — still 88–99.4% on LIBERO. They frame the selection freedom itself as a benchmark weakness any submission could exploit.)

Diagnostic 2: Is the reported gap even statistically meaningful?

Papers report an aggregate success rate and treat any positive gap over the previous best as progress. But significance of a gap depends on task-level and per-instance variance information that the aggregate throws away. The clever move here is to ask: given only two published top-line scores, what can be proven?

For a paired comparison over T tasks with S trials each and per-trial scores in \{0,\dots,R\}, the paired stratified Wald statistic needs the per-instance outcome differences \delta_{t,i}. Those aren't published. So the authors derive two sharp envelopes over all paired outcome tables consistent with the reported aggregates: a rejection-feasibility threshold \delta_\exists (below it, no consistent outcome table can reject the null at \alpha=0.05 — the gap is provably meaningless) and a rejection-guarantee threshold \delta_\forall (above it, every consistent table rejects — the gap is provably significant). In between, the aggregates simply don't determine significance. Appendix C works out the exact integer-programming lower bound and a dynamic-programming upper bound, plus conservative extensions to unpaired evaluations.

Only ~20% of LIBERO and SimplerEnv SOTA claims survive. The authors note a confound honestly: RoboCasa and RoboTwin fare better partly because they have fewer reported results, so scores sit further apart. But the mechanism is the same one that ruins mature leaderboards everywhere — as scores saturate near the ceiling, gaps shrink below what 500 trials can resolve, and the leaderboard turns into noise-ranking. Their prescription is simple and adoptable: benchmarks should publish the minimum gap that counts as significant, and require per-instance outcome files, which let anyone run the actual paired test (which can certify far smaller gaps than aggregates can).

Diagnostic 3: Overfitting that stays inside the training distribution

This is the section with the most surprising numbers, and the design is what makes it sharp. Prior audits (LIBERO-Pro, LIBERO-Plus) perturbed scenes and instructions and showed scores collapse — but their perturbations went outside the training distribution, so a drop could just mean weak generalization, which nobody disputes. Here, every perturbation stays inside the training distribution. Any drop is then overfitting to the benchmark's idiosyncrasies, full stop.

Two flavors. Sample overfitting — fitting the particular fixed test instances — turns out to be nearly absent: redrawing LIBERO's init-state file from the official generator (10,000 rollouts per policy) moves success rates by under a point; fresh CALVIN sequence manifests move ATC by at most 0.11. This mirrors the ImageNet-v2 finding that models hadn't memorized the specific test set.

Distribution overfitting is the killer. CALVIN's ABC→D protocol trains on three scenes with blocks starting anywhere in a pose range, but the released scene-D evaluation fixes the blocks at the same poses every time. Resampling poses from the training range, holding everything else fixed:

Note the pattern: the drop grows with the policy's official score — X-VLA loses 1.03 ATC and 25 points of full-chain success. The leaderboard has been climbing, in part, a fixed-pose artifact. On SimplerEnv's WidowX stack task they find something similar with more texture: three changes that appear in BridgeData V2 training about as often as the official conditions (reversed color instruction, blocks starting on a support block instead of the table, resampled poses). Lower-scoring VLAs (CogACT, InternVLA-M1) drop most on the reversed instruction — suggesting they lean on a language prior rather than reading the instruction. Higher-scoring ones (X-VLA, Dexbotic) handle that fine but collapse on the stacked-support change (X-VLA: 59.7% → 31.3%). The authors' hypothesis is elegant: the benchmark's single fixed camera makes depth/height ambiguous, so policies can learn to assume every block sits on the table plane. Different policies, different shortcuts, same fixed evaluation hiding both.

Diagnostic 4: The score can't tell generalization from proximity

SimplerEnv's premise is that training happens on real BridgeData V2 demonstrations and evaluation happens in a matched simulator — so a high score means transfer across a real sim gap. But the benchmark doesn't restrict where training data comes from. The authors train a 22M-parameter policy per task on 120 scripted demonstrations generated in the same simulator, jittered around the official evaluation grid (±2.5mm XY, ±15° yaw). Result: 91/96 (94.8%), essentially matching X-VLA's 95.8% — the field's reference near-solved score from a 0.9B model with broad pretraining. The gap wasn't crossed; it was deleted. They're explicit that this is an existence proof on a fixed grid, not a fair comparison — but that's the point: the top-line score alone cannot distinguish the two stories. This is the manipulation version of QA-benchmark train/test leakage, and the fix has to be protocol-level: restrict training sources, or require submissions to report train-test proximity.

What to make of it

The meta-result is the inverted correlation between popularity and validity. LIBERO and CALVIN dominate progress claims (Figure 2 of the paper shows their cumulative adoption dwarfing the others) and fail the most diagnostics. RoboCasa and RoboTwin 2.0 — with procedurally generated scenes and heavy domain randomization respectively — resist the shortcut probe (18.8% and ~60% for the probe, far below their SOTAs) and have far higher shares of provably significant claims. The field's evaluation effort is concentrated exactly where it's least informative. The design lesson is stated crisply: benchmark difficulty should come from broad variation in objects and scenes, not from a train-test data gap that unrestricted submissions can simply erase.

If this holds, the practical consequences are immediate. Reviewers can (and should) stop treating sub-noise LIBERO gains as progress; the paper explicitly offers itself as ammunition both for authors who want to skip a broken benchmark and for reviewers questioning a one-point gain. And the ask for per-instance outcome files is cheap and would fix the significance problem overnight.

Things to be skeptical about. The diagnostics are necessary conditions for validity, not sufficient ones — RoboCasa and RoboTwin "fail fewer diagnostics," which doesn't establish that their scores track real-world capability, only that these particular holes aren't found. The creeping-overfitting experiments cover just three CALVIN policies and four SimplerEnv policies, because reproduction requires both released weights and the exact evaluation script, which most papers don't provide (a limitation the authors flag, and itself a damning observation about the field). The significance analysis is conservative for unpaired comparisons. And the probe results involve test-set checkpoint selection, though the disclosed final-checkpoint numbers barely soften the conclusion. None of this threatens the headline claims; it bounds how far they generalize.

There's also a quietly valuable Appendix B on why bitwise reproducibility is the wrong bar for closed-loop evaluation: changing CPU model with the same GPU class flips simulator state at step 0, which propagates through the policy-simulator feedback loop into different pixels by step 54, different actions by step 58, and different success labels by step 88. Contact dynamics amplify float-level noise into outcome-level divergence. Anyone building robot eval infrastructure should read it.

Where to spend your time: Section 5 (creeping overfitting) is the core empirical contribution — the in-distribution perturbation design is the idea most worth stealing. Appendix C is worth a read if you care about what significance can be squeezed from published aggregates alone; the envelope construction is general beyond robotics.