Checkpointing & resume

Long-running eval-sets can periodically snapshot in-progress samples (the Inspect transcript/state plus declared in-sandbox files) to durable S3 storage. If the runner crashes (OOM, a task error, an ungraceful exit), resume the eval-set and its in-progress samples continue from their last checkpoint instead of starting over.

Configure it

Checkpointing only happens with a checkpoint-aware agent — one that ticks Inspect's checkpointer. Most of today's agents don't yet, so you generally have to run (or upgrade to) one that does (see Requirements) to get any checkpoints — until then nothing is snapshotted no matter what the config says.

Checkpointing is off by default. Turn it on — and tune it — with a checkpoint block (see examples/checkpointing.eval-set.yaml); given a supporting agent, this snapshots in-progress samples on a default trigger of one checkpoint every 10 minutes, capturing host state only:

checkpoint:
  enabled: true   # off by default; set true to enable
  trigger:
    type: time   # time | turn | token | manual
    every: 600   # seconds (time) / turns (turn) / tokens (token); omit for manual

Leave the block out (or set enabled: false) to keep checkpointing off — for example on a sandbox that blocks root exec.

Choosing a trigger

• time — wall-clock seconds. Simple; the default.
• turn — every N agent turns. Deterministic regardless of turn length.
• token — every N tokens. Even spacing by cost/progress.
• manual — only when the agent explicitly checkpoints.

Checkpoints fire at turn boundaries, so the effective cadence is max(turn duration, interval) — with very long turns a time/token interval won't fire until the current turn ends. Smaller intervals mean more snapshots (more S3 writes and per-turn overhead); pick the largest interval that bounds an acceptable amount of lost work.

What gets captured in the sandbox

Hawk imposes no default sandbox_paths. A task declares which in-sandbox paths to snapshot on its own samples (Inspect's per-sample checkpoint config), so capture follows the task that knows where its agent works. When sandbox_paths is unset everywhere, checkpoints capture host state only (no in-sandbox files).

The eval-set's checkpoint.sandbox_paths is an eval-wide override, not a fallback: when set, it replaces any task- or sample-declared paths (Inspect merge precedence: eval > sample > task; the whole dict is replaced, not merged key-by-key). Only set it for runs where no task self-declares — otherwise it clobbers each task's own capture paths.

Optional fields:

• sandbox_paths — {<sandbox>: [<absolute paths>]}, an eval-wide override (see above). Paths must be absolute.
• max_consecutive_failures — abort a sample after this many consecutive failed checkpoint writes. Unset (the default) tolerates failures forever, so a sample can run to completion with no usable checkpoint (it just warns on each failed write). Set a small value (e.g. 3) if you'd rather fail fast than discover unprotected work at resume time.
• checkpoints_location — override where checkpoints are written. Defaults to a <eval-log>.checkpoints/ directory beside each eval log in the eval-set's log dir.

Checkpoint artifacts are not cleaned up automatically — the <eval-log>.checkpoints/ restic repos persist in S3 after the run finishes. Expect storage to grow with (samples × snapshots × captured size), and prune old checkpoint directories yourself (e.g. an S3 lifecycle rule on the checkpoint prefix).

Requirements

• A checkpoint-aware agent or solver. Checkpoints fire only for an agent or solver that integrates (ticks) Inspect's checkpointer. metr_agents react is one such agent, but any agent or solver can add the same support; a loop that doesn't tick the checkpointer — or a react build that predates checkpointer support — produces no checkpoints even when checkpointing is enabled.
• Root exec in the sandbox. Inspect injects a restic binary as root to capture sandbox files, so a sandbox that blocks root exec fails the sample at start. Checkpointing is off by default; only enable it on sandboxes that permit root exec.
• A runner image with checkpointing support. Hawk's default runner image bundles a new-enough inspect-ai; a custom --image must ship one too.

Verify it's working

The failure modes above are silent (you simply get no checkpoints), so confirm checkpointing on a short run before relying on it for a multi-day eval:

• hawk trace <eval-set-id> shows checkpoint fires in the Inspect trace.
• A successful checkpoint emits a CheckpointEvent in the sample transcript (visible in the viewer); a failed one logs a warning and a checkpoint_failed info event.
• Checkpoint artifacts appear in S3 under <eval-log>.checkpoints/, one <sample-id>__<epoch>/ subdirectory per sample attempt.

Resume after a crash

hawk eval-set resume <eval-set-id> --secret OPENAI_API_KEY

This restores the saved config from S3 and relaunches at the same id/log dir. Already-completed samples are not re-run; in-progress samples that reached a checkpoint hydrate from it, while samples that never checkpointed restart from the beginning. Resume is repeatable — if the resumed run crashes again, resume again.

Secrets are not stored server-side: pass the same --secret/--secrets-file you launched with, or the resumed run fails on its first API call.

Leftover runner releases

A finished or crashed run often leaves its Kubernetes release behind (the background janitor only reaps it about an hour later, and not at all for some crashed jobs). Resume handles this for you: it reads the run's Kubernetes Job status and, if the Job has finished (a Complete/Failed condition), clears the stale release automatically before relaunching.

Resume only refuses (rather than tearing anything down) when it can't prove the run is safe to clear — so it never clobbers a live run:

• The job is still running → stop it first with hawk stop (graceful) or hawk delete (immediate), then resume.
• The job's state can't be confirmed → clear it explicitly and retry:

hawk delete <eval-set-id>   # removes only the K8s resources; logs & checkpoints in S3 are kept

Resume with a fixed config

To fix something that broke after the agent finished — most commonly a scoring function that crashes on a particular submission — edit the config and resume with it:

hawk eval-set resume <eval-set-id> --config fixed.yaml --secret OPENAI_API_KEY

The expensive agent work is preserved (Inspect resumes for scoring from the agent_complete checkpoint); only the changed parts re-run.

The config must be checkpoint-compatible, and Hawk does not validate this — an incompatible config is accepted and fails at runtime inside Inspect rather than with a clear error. Safe to change: scorers, limits (token_limit/time_limit/cost_limit), retry_attempts, and model/cost config. Not safe: the agents/solvers, epochs, sandbox_paths, or a task package/version that changes sample shape — these break hydration. An explicit <eval-set-id> is required with --config, and any eval_set_id in the supplied config must match it.