operate

Reliability and recovery

Follow a failed user request to its run, find where it went wrong, recover safely, and confirm the fix.

Operationsbeginner0.7.0 Stable
Verified 2026-07-13View sourceReport a docs issue

A reliable Kestrel deployment does more than answer a health check. It preserves session continuity, gives every run a clear outcome, leaves useful history, and recovers without hiding the original failure.

Record the healthy baseline

Before an incident, keep a repeatable request that confirms runner health, authentication, model access, streaming or terminal behavior, and persistence for one known session. Record the Kestrel version and expected result.

Investigate an incident

  1. Identify the person, application route, session, run, and time window.
  2. Capture the terminal event or waiting state before retrying.
  3. Inspect correlation identifiers, traces, artifacts, memory revision, and replay evidence for that same run.
  4. Find the first component that produced an incorrect state.
  5. Apply the least disruptive fix where the problem began.

Confirm recovery

Repeat the original user path with equivalent context. Confirm that the expected result completes, the original history remains available, and authorization, persistence, and cancellation still behave correctly.

Escalate when you cannot identify where the problem began, saved history contradicts current state, access may have been granted incorrectly, or recovery would require destructive data changes.