Runbook: Runner Timeout Storms

A timeout storm is a runner pod so CPU- or memory-starved that it cannot consume its own model response streams. Every in-flight model call then "times out" client-side at the SDK's default deadline — while the LLM gateway sees the same requests complete as healthy, fully-streamed 200s. Each timed-out call wastes the full generation (provider tokens are still consumed) and triggers a retry, amplifying load on an already-starved pod.

On one large production run (METR/hawk#933), 25% of ~297k model calls failed this way — 73,455 client-side timeouts vs 37 genuine 429s — and the state ran invisibly for ~10 hours, three times over.

What it looks like

The signature is bimodal per runner pod:

1. The pod runs clean for hours (0% timeout rate).
2. It flips: 50–100% of all model calls time out, uniformly across every model and provider, sustained for hours.
3. The pod is eventually OOM-killed, restarts clean, and the cycle repeats.

Key discriminator from provider problems: the gateway/provider side is healthy. Gateway CPU is low, SSE keepalives flow, and its traffic log records the "timed-out" requests as complete 200 responses. The failure is entirely in the runner's ability to read the responses.

Detection: metrics and monitors

CloudWatch (all deployments)

The runner emits the storm signals to the Hawk/EvalSet CloudWatch namespace (dimensions inspect_ai_job_id, inspect_ai_created_by; flushed once a minute by hawk.runner.cloudwatch_metrics):

Metric	Meaning
cpu_throttled_pct	% of CFS scheduler periods throttled, sampled every ~5s. The primary starvation signal.
model_calls	Every completed generate() (success or terminal failure).
model_errors (extra dimension error_class: timeout\|rate_limit\|other)	Terminal, retry-exhausted failures, classified from the error text. timeout ≫ rate_limit is the storm signature.
model_attempt_failures	One per failed generate() attempt that will be retried — live during a storm, but carries no cause.

Two Metrics Insights alarms ship with the infrastructure (infra/hawk/runner_pressure_alarms.py, prd stacks): sustained cpu_throttled_pct (default ≥60% for 10 minutes) and terminal timeout volume (default ≥50 per 5 minutes), each per eval-set via GROUP BY — the breaching eval-set id and owner arrive in the SNS contributor notification, on the same webhook config as the stuck-eval monitor (hawk:stuckEvalWebhookUrl).

Datadog (optional, richer breakdowns)

The runner also emits these to Datadog (DogStatsD, tagged inspect_ai_job_id:<eval-set-id> plus per-model/task_name/run_id splits that CloudWatch omits):

Metric	Type	Meaning
hawk.runner.cpu.throttled_pct	gauge	% of CFS scheduler periods throttled since the last tick (~5s). The primary starvation signal — fires during the storm even if no call ever completes.
hawk.runner.cpu.pressure_some_avg10 / pressure_full_avg10	gauge	CPU PSI 10s averages (cgroup v2).
hawk.runner.memory.usage_ratio, pressure_*_avg10	gauge	Memory usage and PSI — reclaim stalls near the limit amplify CPU starvation.
inspect.model.attempt_failures	count	One per failed generate() attempt that will be retried — live during a storm, but carries no cause (timeout vs 429 vs 5xx).
inspect.model.retry_wait	histogram	Backoff seconds scheduled per retry.
inspect.model.calls	count	Every completed generate() (success or terminal failure).
inspect.model.errors (tag error_class:timeout\|rate_limit\|other)	count	Terminal, retry-exhausted failures, classified from the error text. timeout ≫ rate_limit is the storm signature.

Two Datadog monitors ship with the infrastructure (infra/datadog/runner_pressure_monitor.py):

• "Hawk runner is CPU-starved" — hawk.runner.cpu.throttled_pct above threshold (default 60%) for 10 sustained minutes, per eval set.
• "Hawk eval set is failing model calls on timeouts" — more than N (default 50) terminal error_class:timeout failures in 15 minutes, per eval set.

Thresholds are deployment-tunable via Pulumi config: datadog:runnerCpuThrottledPctThreshold, datadog:runnerTimeoutErrorsThreshold (see Pulumi.example.yaml).

Note the two timeout alarms share a default of 50 but not a sensitivity: the Datadog monitor fires on 50 timeouts across 15 minutes, while the CloudWatch alarm needs ≥50 in each of two 5-minute periods (≈100+/15m). This is intentional — CloudWatch is the always-on pager and tuned quieter; Datadog corroborates and catches slower burns.

Known gap: a timeout that is retried and eventually recovers only shows up in attempt_failures/retries without a cause — inspect_ai does not surface the retry cause to hooks. During a real storm this doesn't matter in practice: the CPU monitor fires on the root cause within minutes, and sustained storms exhaust retries into classified terminal errors.

Diagnosis flow

When either monitor fires (or a run looks stuck with mass retries):

1. Confirm starvation. Check hawk.runner.cpu.throttled_pct (Datadog) or Hawk/EvalSet cpu_throttled_pct (CloudWatch) and CPU PSI for the eval set. Pegged values mean the runner is starved — the timeouts are a symptom, not the disease.
2. Classify the errors. Break inspect.model.errors down by error_class. timeout-dominated → starvation. rate_limit-dominated → provider throttling (a different problem; see the middleman dashboard's rate-limit row).
3. Check memory pressure. hawk.runner.memory.usage_ratio climbing toward 1.0 with rising pressure_full_avg10 means reclaim stalls are contributing — and an OOM kill (with a full sample-retry cycle) is coming.
4. Verify the gateway is healthy. The gateway traffic log should show the same requests completing with 200s. If the gateway is also erroring, this is not a runner starvation storm.
5. Live inspection (see Debugging Evaluations): hawk status <eval-set-id> for pod state, hawk stacktrace <eval-set-id> for a py-spy dump showing where the runner is stuck.

Remediation

• Raise the runner's CPU for the eval set: runner.cpu in the eval-set config (Kubernetes quantity, e.g. "4"). Large eval sets with many concurrent samples/sandboxes need proportionally more runner CPU.
• Slow the sandbox ramp. Bursts of concurrent sandbox helm installs can transiently starve the runner (this reproduced the failure on demand upstream). Lower sandbox-creation concurrency if configured.
• Raise runner.memory if memory pressure is the amplifier — an OOM cycle re-pays the entire ramp and retries in-flight samples.
• Prefer a controlled restart over hours of spinning. If a pod is deep in the degraded state, hawk stop / resume wastes less than letting a 100%-timeout pod grind until its OOM kill.

Related issues

• METR/hawk#933 — this incident and the detection layer described here.
• METR/hawk#930 / #931 — the prevention layer: sizing pod-op and helm-install concurrency from the pod's actual CPU limit (PR #946). Prevention makes storms rare; this page is for when one happens anyway.