Same 100 episodes.
16 points better.

01What we tested and how

Dataset

robomimic v1.5 · square (NutAssemblySquare) · low-dim

Policy

BC-RNN · 600 epochs · 5 seeds per condition

Compute

1× AWS g6.2xlarge · 20 runs · ~12 USD

Data

robomimic square: a Franka Panda picks up a square nut and threads it onto a peg, in simulation, 20 Hz, low-dimensional observations (end-effector pose, gripper, object state). The multi-human set holds 300 successful demonstrations from six operators of varying skill, two each labelled better / okay / worse by the dataset authors. The labels were hidden from the selection and opened only afterwards, to report what each selection actually contained.

Selection metrics

Six per-episode quality metrics from the KP Digital QC pipeline: length excess over what the starting geometry predicts, path efficiency measured through the object, stop ratio, direction reversals, gripper cycles, approach retries. Each is z-scored over the 300 episodes and averaged; lower is better. Jerk was excluded because on this task the expert operator has the higher jerk (decisive motion), which we learned in an earlier test.

Four conditions

100 training episodes each, same policy (BC-RNN, robomimic paper settings, 600 epochs), same hyperparameters, five seeds each. Evaluation every 50 epochs, 50 rollouts from random object placements, 500-step horizon. The primary metric is the last checkpoint, not the best one, to avoid checkpoint-selection bias.

Note the first row. With no labels at all, ranking operators by our metrics recovered the dataset authors’ “better” operators exactly. The full six-operator ordering also matched: better, better, okay, okay, worse, worse.

Pre-registered before the first run

The selection rule above, the decision thresholds in section 3, and numerical expectations for each condition (46 / 41 / 35 / 31 percent). None of these was changed after training started.

02Results

By seed

03Verdict against the pre-registered rules

• Operator-level selection works. With the same budget of 100 episodes, the selected set trains a policy to twice the success rate of a random set. The selection used operator identity plus our metrics, and no human judgement of skill.
• Episode-level selection, on its own, does not. Ranking episodes individually reached 67% expert purity, and that was not enough to move the policy. The practical lesson: keep operator identity in the metadata. On our collection floor it is always recorded.
• The gap between the best and worst 100 is 22 points. The same dataset contains both. A buyer who cannot tell them apart is paying for both.
• Seed spread is wide. G1 ranges from 16% to 44% across seeds. That is why we report five seeds and pooled intervals, and why a single-seed result would not have been enough to claim anything.

04Honest notes

• Absolute rates are low and expected to be. 100 episodes is one third of the benchmark, we train 600 epochs rather than the paper’s 2,000, and we report the last checkpoint rather than the best. All four conditions share these handicaps; the comparison between them is what the test measures. Do not read 30% as the ceiling of this data. The same protocol on all 270 episodes gave 34%; on 180 episodes from one proficient operator, 59%.
• Our pre-registered expectations were too high. We predicted 46 / 41 / 35 / 31 and observed 30 / 17 / 15 / 9. The differences between conditions came out close to prediction; the absolute level did not.
• Validation logging was disabled. The training framework refuses a validation split that overlaps the training split, and any 100-of-300 selection overlaps the benchmark’s 30-episode validation set. In this setup validation loss is used only for logging, never for checkpoint choice or early stopping, so reported metrics are unaffected. Recorded in the decision log at the time it was decided.
• One benchmark, one task, simulation, low-dimensional observations. The effect size will differ on real robots and on image observations. The direction and the ordering of conditions are what we claim.
• Six operators, two selected. Operator-level selection needs more than one operator to rank. With a single operator per dataset the method reduces to the episode-level case, which did not work here.
• Preceded by a failure. An earlier test with the same data applied a filter that removed the 21% of episodes flagged by relative thresholds. It changed nothing (31% vs 34% vs 35% for filtered, full, and random of equal size). That result is what led to selecting against a reference instead of trimming a distribution’s tail.

05Throughput

The selection above is only useful if it runs at the scale of a real corpus. On 28 September 2026 we ran the KP Digital integrity checker over every public LeRobot dataset we had downloaded from Hugging Face, on one 32-core server (AWS c7i.8xlarge, 61 GB memory), and timed it.

• 845 Datasets
• 524,351 Episodes
• 213,771,693 Frames
• 2,881 h Demonstration time covered
• 200 s Scan wall-clock, all 845 datasets
• 62 s Transfer from storage and unpacking (not included above)

Result totals vs the earlier single-process run: identical (3,089 defective episodes, 0.74% of the 419,284 in healthy datasets).

• What the checker does per episode. NaN and range checks, timestamp monotonicity, frozen state while commands move, impossible state jumps, duplicated frames, and per-dataset stream health (state updating slower than the frame rate, constant channels, systematic stalls). The same code that produced the quality metadata used in the selection test.
• How the time splits. The 19 largest datasets (over 300 MB each) took 150 of the 200 seconds because they must run at low concurrency; the remaining 826 datasets took 50 seconds at full concurrency. Splitting large datasets by episode would cut the total further. Not done yet.
• Honest note. The first two attempts at full concurrency ran out of memory: one dataset needs up to 15.6 GB in memory, about 15× its file size. Concurrency was then tiered by dataset size. The 200 seconds is the third, successful run, timed end to end, and its totals match the original run exactly.

06What this means if you buy or build data

• Ask who operated, per episode. Operator identity is the single most predictive field in this test. If a dataset arrives without it, the best a filter reached here was 67% purity, and that did not move the policy.
• Selection, not filtering. Removing a tail did nothing; choosing against a quality reference doubled success. The metrics are the same in both cases. What they are anchored to is what matters.
• Demand the negative results. A QC vendor who has never published a test that failed has not tested.

07Reproduce it

The deliverable archive holds the selection script, the QC metadata it reads, the 20 training configs, the evaluation CSV, the timestamped decision log, and the scan logs with per-tier timings. Selection is label-blind; the label file is opened only for the composition table.

# Selection: writes the four filter keys into a copy of the dataset and prints the composition table
python make_filters_exp2.py --qc_meta qc_meta_square_mh.parquet --labels labels_square.parquet \
                            --hdf5 square_mh_low_dim_v15_qc.hdf5

# Training (robomimic 0.5, robosuite 1.5.1, mujoco 3.2.3), 20 runs, 3 concurrent, then tables and verdict
bash run_exp2.sh && python collect_exp2.py && python build_exp2_verdict.py

# Public-data scan: any folder of LeRobot v2/v3 datasets, dataset-level parallelism
python scan_lerobot.py --data ./data --out results --jobs 32 && python finalize_scan.py results

Data: robomimic, Mandlekar et al., “What Matters in Learning from Offline Human Demonstrations for Robot Manipulation”, CoRL 2021. MIT licence.

Report generated 28 September 2026 from eval/results_exp2.csv and results_server/final_total.json.