
A new robotics preprint argues that success-checking code can misread whether a simulated task has actually been completed. Its SimAudit pipeline found 61 distinct checker defects across 49 tasks in RoboTwin 2.0, RoboCasa365 and RoboDojo, according to the authors. Some accepted incomplete executions; others rejected valid alternatives.
The paper was submitted November 18. Its full methods and results distinguish phantom success from silent failure. That distinction prevents an overly simple conclusion that all benchmark scores were inflated.
The judge can be wrong in either direction
The authors report 40 phantom successes and 21 silent failures. Examples include checking a gripper command instead of the physical finger state, accepting loose geometric conditions, or demanding an orientation the task instruction never specified.
SimAudit proposes candidate situations from the instruction and checker code, recreates them in simulation and runs the unmodified checker. The visual-review stage rejected about two-thirds of 183 candidates that had produced the predicted verdict. Two annotators independently confirmed the retained defects. These are reported research procedures; we have not rerun them.
Our analysis: a benchmark needs negative examples too
Passing a demonstration shows that a checker recognizes one successful path. It does not show that the checker rejects an incomplete attempt or accepts a different valid solution. A reliable evaluator needs both kinds of test.
That lesson applies beyond robots. Whenever software compresses a complicated outcome into a pass or fail, the definition of success becomes part of the experiment. A policy can be judged against the wrong objective even when the simulator executes its actions faithfully.
It is also important to distinguish a flaw that can be constructed from one a policy commonly reaches. The paper therefore rescored released policies' rollouts with repaired checkers instead of stopping at adversarial examples.
Changed scores did not mean a new ranking
The largest reported share of changed episode verdicts was 9.1 percent for one evaluated policy. Scores moved in different directions across the benchmarks, and the tested rankings remained unchanged. That is narrower than a claim that the study overturned all existing leaderboards.
These findings concern three simulated benchmark implementations and the authors' selected repair and review process. They do not establish real-world robot safety or prove that every unreported checker is correct. A larger audit or another interpretation of a task could reveal additional issues.
The authors say they release auditing material and repaired checkers, but we did not verify a direct public implementation repository from the source links reviewed here. We therefore link the paper and its examples rather than invent a code destination. This is a preprint and a source review, not independent validation of its results.
Original source and implementation review with labeled editorial analysis. No model inference, installation, paid service, API benchmark or independent security/robotics test performed.






