Evaluation
Find out whether your robot learned the skill, and how well.
Expert-defined criteria, reserved test scenarios and a report on where each policy version breaks down.
Policy attemptIllustrative
Basic completion checks
- Wire in terminalPass
- Pull checkPass
- Screw seatedPass
Expert-defined quality criteria
- Insulation under the clampFail
Insulation is trapped under the clamp. The connection fails the agreed assembly criteria.
How it runs
Agree the test before anyone trains for it.
We run it and report, or package it for your team to run in-house. Your choice.
- Agree
Rubric and scenarios
Set with your team before any data is collected.
Rubric- Strip length in range
- No insulation under clamp
- Pull check
- Reserve
Kept out of training
Test scenarios never appear in the training data.
TrainingReserved for test - Run
Every version, same test
We run each policy version on the reserved scenarios.
Reserved scenariosCurrentNext version - Review
Expert-defined criteria
Experts define what correct means. Each criterion is checked automatically or by review.
Automatic3 / 3 passExpertInsulation under clamp - Report
Where it breaks down
Results by criterion and scenario, across versions.
Reserved scenarios- Current policy
- After training
Evaluation
Agree the test before you collect the data.
Results by criterion and version, on a test set that is never trained on. Expert labels and reward design are available as separate work.