Deployment troubleshooting

Symptom	Check first	Verification after the fix
Runner health fails	Process, service URL, and deployment logs	Health response reports compatible versions and capabilities
Runner is healthy but commands fail	Service token and command metadata	`runner.ping` succeeds with the intended tenant context
Application route returns an error	Server route configuration and correlation id	Direct runner check and application route both succeed
Stream closes unexpectedly	Client disconnect, forwarded abort signal, and terminal event	Disconnect produces `run.cancelled`; normal request completes
Wrong user or organization context	Server-side authentication resolver	Allowed account succeeds and denied account receives no protected data
Model cannot be selected	Gateway visibility, approval, deployment readiness, and lease	Short request completes through the requested model

Start from the symptom

Record the exact visible failure, actor, environment, request/correlation identifiers, time, and deployed revision before restarting or redeploying anything.

Runner checks

Bash

curl -sS "$KESTREL_RUNNER_SERVICE_URL/health"
 
curl -sS "$KESTREL_RUNNER_SERVICE_URL/commands" \
  -H "Authorization: Bearer $KESTREL_RUNNER_SERVICE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "cmd-operator-check",
    "type": "runner.ping",
    "metadata": {
      "actor": {
        "actorId": "operator-check",
        "actorType": "operator",
        "tenantId": "test"
      },
      "tenantId": "test"
    },
    "payload": { "nonce": "operator-check" }
  }'

Preserve the request, correlation, session, and run identifiers before changing the deployment. Use Reliability and recovery to confirm that the original user path works afterward.

Identity and auth

Prove server-owned actor/tenant resolution and an allowed and denied request. A runner token or provider credential must never be accepted from the browser.

Provider and profile capability

Compare the canonical profile, provider-registry revision, model readiness, required capabilities, qualification, lease, and budget evidence.

Database and migrations

Inspect the deployed schema and pending list. Stop when compatibility or backup evidence is missing; do not replay already-applied migrations to make the list look current.

Queue and worker delivery

Check queue ownership, stuck work, retry evidence, the selected image tag on each affected Machine, and the terminal event emitted for the affected run.

Vercel and Fly deployment identity

Compare the user-facing Vercel deployment and each affected Fly Machine's provider record with the operator's release notes. Diagnose each target independently.

Execution policy and external approvals

Inspect the resolved profile, tool descriptor, network/OCI policy, pending request, and exact action approval before treating a denial as infrastructure failure.

Roll back or fix forward

Roll Vercel back through its native provider operation and redeploy each selected Machine with its recorded prior tag only when migration compatibility is proven. Otherwise stop and fix forward without destructive data reversal.

Confirm the user path

Repeat the smallest original path—from authentication through terminal result or artifact—and confirm persisted evidence, hosted identity, and operator controls agree.