Debugging Stuck Evaluations¶

This guide documents debugging techniques for evaluations that aren't progressing.

Quick Diagnosis Checklist¶

Verify authentication: hawk auth access-token > /dev/null || echo "Run 'hawk login' first"
Watch live status: hawk watch <eval-set-id> --no-follow — per-task / per-sample phase, retries, limits, scores, and pod/scheduling trouble in one snapshot. Usually tells you why it's stuck without touching logs (see Live Status).
Check job status: hawk status <eval-set-id> — JSON report with pod status, logs, metrics
View logs: hawk logs <eval-set-id> or hawk logs -f for follow mode
Read the Inspect trace log: hawk trace <eval-set-id> shows the runner's in-flight actions (an enter with no matching exit = currently hanging). Start with the default tail (-n); don't print --full --raw — the full raw trace can be very large. When you need the whole thing, redirect to a file and let inspect summarize it: hawk trace <id> --full --raw > trace.log then inspect trace anomalies trace.log (--raw keeps the JSON-lines format the inspect trace tooling expects). Only works while the runner pod is running. Add -f to follow live.
Capture a stack trace: hawk stacktrace <eval-set-id> — py-spy dump of the runner's live thread stacks; shows exactly where the runner is blocked right now. Only works while the runner pod is running.
List samples: hawk list samples <eval-set-id> — see which samples completed/failed
Get transcript: hawk transcript <sample-uuid> — view conversation for a specific sample
Check buffer in S3: Download .buffer/ segments to analyze eval state
Test API directly: Reproduce the error outside Inspect to isolate the issue

System Architecture¶

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   hawk CLI   │────>│  API Server  │────>│    Helm      │
│ (eval-set)   │     │   (FastAPI)  │     │  (releases)  │
└──────────────┘     └──────────────┘     └──────────────┘
                                                │
                                                v
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  S3 Logs     │<────│ Runner Pod   │────>│ Sandbox Pods │
│ (.eval,      │     │ (runner ns)  │     │ (eval-set ns)│
│  .buffer/)   │     └──────────────┘     └──────────────┘
└──────────────┘            │
                            v
                     ┌──────────────┐     ┌──────────────┐
                     │  Middleman   │────>│ Provider APIs│
                     │  (auth proxy)│     │              │
                     └──────────────┘     └──────────────┘

Live Status (`hawk watch`)¶

hawk watch is the fastest first look at a stuck run. It reads the /monitoring/jobs/{id}/eval-set-status endpoint — the same per-task / per-sample data the web viewer shows — and renders it as a live dashboard.

hawk watch <eval-set-id>               # live-streaming dashboard (Ctrl-C to stop)
hawk watch <eval-set-id> --no-follow   # one-shot snapshot, then exit
hawk watch <eval-set-id> --json        # raw EvalSetStatus payload for parsing

Top-line job status: Queued / Running / Importing samples / Stuck / Complete / Failed. Stuck means Kubernetes reported a trouble reason while pods are still active — it isn't progressing on its own.

⚠ trouble_reason (eval-set-level or per-task) is the most actionable signal:

Trouble reason	Meaning	Resolution
`runner out of memory` (OOMKilled, runner pod)	Runner exhausted memory	Increase runner memory limit
`image pull failing` (ImagePullBackOff / ErrImagePull)	Bad/inaccessible runner or sandbox image	Check image tag / registry auth
`container crash looping` (CrashLoopBackOff)	Runner container keeps dying	Check `hawk logs` for the crash
`pods can't be scheduled` (Unschedulable)	No K8s capacity past the grace period	Wait for autoscale, or check cluster capacity

OOMKilled / CrashLoopBackOff on a sandbox pod is per-sample — Inspect retries it automatically. Only runner-pod trouble is reported at the job level.

waiting for Kubernetes — N sandbox pods not yet scheduled — samples whose sandbox pods are still Pending (not in the Inspect buffer yet). A few briefly is normal; many stuck for a while points at cluster capacity or image-pull trouble (the per-pod reason/detail is in --json under waiting_samples).

Per-sample lines (shown for notable samples — running, errored, hit a limit, or being retried):

phase: waiting / init / running / scoring / completed / errored / limit / canceled
N retries — Inspect is re-running the sample. A few is fine; one stuck retrying is the thing to investigate.
limit: <token|time|message|...> — the sample halted on a limit, not a bug. Raise the limit if unexpected.
score <x> — latest score once scored.
--json additionally carries per-sample last_events (the in-flight event has pending: true), so you can see what a running sample is doing right now (waiting on a model call, a tool, the sandbox), plus error, total_tokens, and timings.

(status degraded: ...) — a data source (pod_status / metrics / tasks) failed but the rest of the payload is still usable; not the eval's problem.

Per-Sample Execution Timeline¶

When one sample is slow or stuck and you need to know where its time went, query the per-sample timeline (HWK-27): GET /meta/samples/{sample_uuid}/timeline. It returns the X-Ray span waterfall the runner and middleman already emit — model calls, tool calls, middleman routes, provider requests, scoring, setup — so a single long span (a hung tool call or provider request) stands out.

There's no CLI command yet (the web view is a later phase), so curl it:

TOKEN=$(hawk auth access-token)
curl -s -H "Authorization: Bearer $TOKEN" \
  https://api.inspect-ai.internal.metr.org/meta/samples/<sample-uuid>/timeline \
  | jq '.spans | sort_by(-.duration_ms) | .[0:10] | .[] | {name, category, duration_ms}'

The response is {sample_uuid, eval_set_id, eval_id, spans, truncated}. Each span carries span_id, parent_id (build the tree via parent_id → span_id), name, category, kind, start_ms, end_ms, duration_ms, and attributes. Categories: setup, run, sample, execution, scoring, model, tool, middleman, provider, heartbeat, other.

An empty spans list is not an error — it means tracing is off for that deployment, the eval predates span retention, or it was never traced. The query is on-demand (Logs Insights latency, a few seconds) with no cache in v1.

Common Error Patterns¶

API 500 Errors with Retries¶

What it looks like:

Error code: 500 - Internal server error
Retry attempt 45 of 50 failed. Waiting 1800 seconds before next retry.

Diagnosis:

Download the .buffer/ from S3 to find the failing request
Test the request through Middleman: curl https://middleman.internal.metr.org/...
Test directly against the provider API to isolate whether it's a Middleman issue

Note

500 errors are NOT token limit issues. Token limits return 400 errors.

Token/Context Limit Errors (400)¶

Error code: 400 - invalid_request_error

This IS a token limit issue. Check message count, configured token_limit, and model context window.

Retry Logs¶

Retry log messages include a context prefix identifying the sample:

[nWJu3Mz mmlu/42/1 openai/gpt-4o] -> openai/gpt-4o retry 3 (retrying in 24 seconds) [RateLimitError 429 rate_limit_exceeded]

The prefix format is [{sample_uuid} {task}/{sample_id}/{epoch} {model}].

Tip

The OpenAI SDK does not show the actual error in its retry messages. You must test with curl directly to see the real error.

Pod UID Mismatch¶

Pod UID mismatch: expected abc123, got def456

The sandbox pod was killed and restarted. There's nothing to do — Inspect will automatically retry the sample.

OOMKilled Pods¶

Check pod status via hawk status or:

kubectl describe pod -n <runner-namespace> <pod-name> | grep -A5 "Last State"

Look for OOMKilled in the termination reason.

Accessing Eval State¶

From S3¶

# List eval set contents
aws s3 ls s3://<bucket>/evals/<eval-set-id>/

# Download buffer for analysis
aws s3 sync s3://<bucket>/evals/<eval-set-id>/.buffer/ /tmp/buffer/

# Download completed .eval files
aws s3 sync s3://<bucket>/evals/<eval-set-id>/ /tmp/results/ --exclude ".buffer/*"

Reading .eval Files¶

.eval files are zip archives. Read them using the inspect_ai library:

from inspect_ai.log import read_eval_log

log = read_eval_log("path/to/file.eval")
for sample in log.samples:
    print(sample.id, sample.score)

Sample Buffer Analysis¶

The .buffer/ directory contains SQLite databases with eval state.

Check overall progress:

SELECT COUNT(*) as total_events FROM events;

SELECT json_extract(data, '$.state') as state, COUNT(*)
FROM samples GROUP BY state;

Detect if eval is progressing (FAIL-OK pattern):

SELECT
  id,
  CASE WHEN json_extract(data, '$.error') IS NULL THEN 'OK' ELSE 'FAIL' END as status
FROM events
WHERE json_extract(data, '$.event') = 'model_output'
ORDER BY id DESC LIMIT 50;

FAIL, OK, FAIL, OK... = Progressing (transient errors, retries succeed)
FAIL, FAIL, FAIL... = Stuck (something changed, now all fail)

Find pending events (stuck API calls):

SELECT COUNT(*) FROM events WHERE json_extract(data, '$.pending') = 1;

Direct API Testing¶

Test through Middleman to isolate whether the issue is Inspect AI, Middleman, or the provider.

TOKEN=$(hawk auth access-token)

# Test Anthropic
curl --max-time 300 -X POST https://middleman.internal.metr.org/anthropic/v1/messages \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "max_tokens": 100,
    "messages": [{"role": "user", "content": "Say hello"}]
  }'

# Test OpenAI-compatible APIs
curl --max-time 300 -X POST https://middleman.internal.metr.org/openai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Say hello"}],
    "max_tokens": 100
  }'

Interpretation:

Result	Meaning
All tests pass	Issue is content-specific or Inspect AI bug
Middleman fails, direct to provider works	Middleman issue
Both fail	Provider API issue
Large request fails with 400	Token limit
Large request fails with 500	API bug with large context

Early Warning Signs¶

HTTP Retry Count Growth¶

Task progress logs include an "HTTP retries" counter:

{"message": "Task: search_server, Steps: 12/12 100%, ... HTTP retries: 5,710", ...}

Tasks can complete successfully despite thousands of retries. A growing retry count indicates API instability, but isn't fatal until retries stop succeeding entirely.

Severity calculation: Retry count x wait time = stuck duration. E.g., 45 retries x 1800s = 22+ hours stuck.

Command Cheatsheet¶

Hawk CLI¶

hawk watch <eval-set-id> --no-follow         # Live per-task/per-sample status snapshot
hawk status <eval-set-id>                    # JSON monitoring report
hawk logs <eval-set-id>                      # View logs
hawk logs -f                                 # Follow logs live
hawk trace <eval-set-id>                     # Inspect trace log tail (live runner pod only)
hawk trace <eval-set-id> --full --raw > trace.log  # Full raw trace (can be very large) to file; then: inspect trace anomalies trace.log
hawk stacktrace <eval-set-id>                # py-spy stack dump of the runner (live runner pod only)
hawk list samples <eval-set-id>              # List samples with status
hawk transcript <sample-uuid>                # View sample conversation
hawk delete <eval-set-id>                    # Delete eval and cleanup
hawk eval-set <config.yaml>                  # Start/restart eval

S3 Access¶

aws s3 ls s3://<bucket>/evals/<eval-set-id>/
aws s3 sync s3://<bucket>/evals/<eval-set-id>/.buffer/ /tmp/buffer/

Kubectl (Advanced)¶

kubectl get pods -n <runner-ns> | grep <eval-set-id>   # Find runner pod
kubectl logs -n <runner-ns> <pod-name> --tail=200      # Pod logs
kubectl get pods -n <eval-set-id>                      # Sandbox pods
kubectl describe pod -n <runner-ns> <pod-name>         # Full pod details

Escalation Checklist¶

If you can't resolve the issue:

Gather evidence: hawk status output, error patterns, sample buffer state, failing request payloads
Determine scope: One eval or multiple? One task or all? One model or all?
Check external status: Provider status pages, AWS status, internal monitoring dashboards
Document timeline: When did the eval start? When did it get stuck? Last successful operation?