A reliable Kestrel deployment does more than answer a health check. It preserves session continuity, gives every run a clear outcome, leaves useful history, and recovers without hiding the original failure.
Record the healthy baseline
Before an incident, keep a repeatable request that confirms runner health, authentication, profile/provider readiness, model access, migrations, workers/images, streaming or terminal behavior, and persistence for one known session. Record version, revision, image digests, and expected result.
Durable conversation and terminal delivery
An accepted run retains its conversation identity and produces one terminal outcome. A transport disconnect may interrupt presentation, but it must not erase the run or cause a second logical operation.
Policy-bound recovery
Recovery actions come from the runtime registry and policy. Bind the operator's selection to the exact pending request and option identifier, and revalidate authority before resuming.
Compaction and replay integrity
Compaction may reduce model-visible context but cannot rewrite committed messages, terminal outcomes, approvals, budgets, or evidence. Replay uses the saved canonical records and labels provider-dependent behavior honestly.
Local Core and worker self-healing
Readiness and build identity must agree before automated restart. Content-aware restart may replace an unhealthy or stale process; it must not hide a schema, provider, or policy mismatch behind repeated restarts.
Investigate an incident
- Identify the person, application route, session, run, and time window.
- Capture the terminal event or waiting state before retrying.
- Inspect correlation identifiers, traces, artifacts, memory revision, and replay evidence for that same run.
- Find the first component that produced an incorrect state.
- Apply the least disruptive fix where the problem began.
Confirm recovery
Repeat the original user path with equivalent context. Confirm that the expected result completes, the original history remains available, and authorization, persistence, and cancellation still behave correctly.
Escalate when you cannot identify where the problem began, saved history contradicts current state, access may have been granted incorrectly, or recovery would require destructive data changes.