Ruhroh runs Kestrel evaluation scenarios, records the results, and compares one run with another. Use these checks before releasing runtime, prompt, or model changes that could alter user-visible behavior.
Evaluation model
Runtime evaluators enforce one run's delivery, policy, terminal, recovery, or quality contract. Ruhroh performs repeatable comparative evaluation across scenarios, versions, prompts, or providers. Keep those roles separate in evidence.
Core commands
pnpm run ruhroh:validateThis command validates the Ruhroh configuration recorded by Kestrel. It does not execute an evaluation.
What the gate validates
The release check confirms that:
- each declared behavior maps to a maintained scenario, suite, or Kestrel test
- scenario, instruction, and released package versions match the recorded evaluation
- Ruhroh accepts the scenarios, suite, benchmark targets, and adapter dry run
These checks do not call a live model provider. Run live benchmarks separately when a release changes model-dependent behavior.
Runtime evaluation policy
Record the evaluator identity and version, trigger, required inputs, delivery consequence, and evidence with the run. An evaluator may block delivery only through its declared policy; assistant prose cannot override it.
Calibration datasets and assets
Version scenario inputs, expected contracts, calibration datasets, prompts, provider/profile selection, and supporting artifacts. Historical assets remain immutable comparison evidence.
Delivery control and terminal outcomes
Evaluation progress is not a terminal result. Record whether a gate admitted, blocked, or requested recovery and preserve the run's actual completed, failed, cancelled, or waiting outcome separately.
Historical comparison records
The files under evals/migration/parity/ preserve earlier comparison results.
They record the scenario and instruction versions, outcome, cancellation state,
and failure classification used during the evaluation transition.
Use these files for historical comparison. Add new behavior checks as current Ruhroh scenarios or Kestrel tests rather than editing old results.
Workflow
- change the Kestrel runtime or an evaluation definition
- run
pnpm run ruhroh:validate - inspect any failing Ruhroh validation output
- update the scenario, suite, or Kestrel test that describes the intended behavior
- rerun the check and review the comparison before release