Ruhroh runs Kestrel evaluation scenarios, records the results, and compares one run with another. Use these checks before releasing runtime, prompt, or model changes that could alter user-visible behavior.
Core commands
pnpm run ruhroh:validateThis command validates the Ruhroh configuration recorded by Kestrel. It does not execute an evaluation.
What the gate validates
The release check confirms that:
- each declared behavior maps to a maintained scenario, suite, or Kestrel test
- scenario, instruction, and released package versions match the recorded evaluation
- Ruhroh accepts the scenarios, suite, benchmark targets, and adapter dry run
These checks do not call a live model provider. Run live benchmarks separately when a release changes model-dependent behavior.
Historical comparison records
The files under evals/migration/parity/ preserve earlier comparison results.
They record the scenario and instruction versions, outcome, cancellation state,
and failure classification used during the evaluation transition.
Use these files for historical comparison. Add new behavior checks as current Ruhroh scenarios or Kestrel tests rather than editing old results.
Workflow
- change the Kestrel runtime or an evaluation definition
- run
pnpm run ruhroh:validate - inspect any failing Ruhroh validation output
- update the scenario, suite, or Kestrel test that describes the intended behavior
- rerun the check and review the comparison before release