Running Evaluations¶

The Eval Set Config¶

An eval set config is a YAML file that defines a grid of tasks, agents, and models to evaluate. Hawk runs every combination.

Here's a minimal example:

tasks:
  - package: git+https://github.com/UKGovernmentBEIS/inspect_evals
    name: inspect_evals
    items:
      - name: mbpp

models:
  - package: openai
    name: openai
    items:
      - name: gpt-4o-mini

limit: 1  # optional: cap the number of samples

Submit it:

hawk eval-set config.yaml

Adding Agents¶

agents:
  - package: git+https://github.com/METR/inspect-agents
    name: metr_agents
    items:
      - name: react
        args:
          max_attempts: 3

Eval Parameters¶

These fields are passed through to inspect_ai.eval_set():

Field	Description
`limit`	Maximum samples to evaluate
`time_limit`	Per-sample time limit in seconds
`message_limit`	Maximum messages per sample
`epochs`	Number of evaluation epochs
`eval_set_id`	Custom ID (auto-generated if omitted)
`metadata`	Arbitrary metadata dictionary
`tags`	List of tags for organization
`log_images`	Whether to log base64-encoded images in the eval log. Omit to use inspect-ai's default (`true`).

For the complete list of fields, see the Eval Set Config Reference or the Inspect AI docs.

Secrets and API Keys¶

Pass environment variables to your eval runner with --secret or --secrets-file:

# From your environment
hawk eval-set config.yaml --secret MY_API_KEY

# From a file
hawk eval-set config.yaml --secrets-file .env

# Both
hawk eval-set config.yaml --secrets-file .env --secret ANOTHER_KEY

By default, Hawk routes model API calls through its managed LLM proxy (supporting OpenAI, Anthropic, and Google Vertex). To use your own API keys instead, pass them as secrets and disable the proxy's token refresh:

runner:
  environment:
    INSPECT_ACTION_RUNNER_REFRESH_URL: ""

You can also declare required secrets in your config to catch missing credentials before the job starts:

runner:
  secrets:
    - name: DATASET_ACCESS_KEY
      description: API key for dataset access

AWS Secrets Manager¶

If your deployment enables it, the runner can fetch a secret's value from AWS Secrets Manager instead of you passing it on the command line. Set type: aws-secrets-manager and the runner reads <default prefix><name> at startup, injecting the value as the environment variable name:

runner:
  secrets:
    - name: HF_TOKEN
      type: aws-secrets-manager
      description: Hugging Face token fetched by the runner at startup

With the default namespace (inspect-tasks/), that reads the secret inspect-tasks/HF_TOKEN.

To read a secret whose name differs from the environment variable — for example when several teams need their own value behind a shared variable name — add secret_name. It resolves under the same default prefix, and the task still sees HF_TOKEN:

runner:
  secrets:
    - name: HF_TOKEN
      type: aws-secrets-manager
      secret_name: team-x/HF_TOKEN

Use arn instead when the secret lives outside the default prefix entirely; it takes a full ARN with no wildcards. secret_name and arn are mutually exclusive.

Your deployment restricts which secrets a runner may read, so a secret_name or arn resolving outside the allowed namespace is rejected before the job starts. hawk local fetches these secrets with your own AWS credentials.

Additional Packages¶

Install extra Python packages into the runner's virtualenv:

packages:
  - git+https://github.com/some-org/some-package

Private GitHub repos work automatically if Hawk's GitHub token has access. Both git@github.com: and ssh://git@github.com/ URL formats are supported and converted to HTTPS internally.

Overriding inspect-ai¶

Eval-set jobs install the same inspect-ai that Hawk itself runs, so a job can't silently pick up a version Hawk hasn't been tested against. Naming inspect-ai in packages: overrides Hawk's pin:

packages:
  - inspect-ai==<version>

(Nothing in an eval-set job uses inspect-scout — it only applies to scans, see below — so naming it here just installs an unused package. It doesn't override a pin, because there is no inspect-scout pin on this path.)

For scans: the same override mechanism also covers inspect-scout, since scan jobs install and pin both packages. Naming either in a scan config's packages: drops that package's extra and its pin. Scout's exact version doesn't come from detected install metadata the way inspect-ai's does — the runner image never installs inspect-scout directly, so there's nothing to introspect there — it comes from the exact pin Hawk's own pyproject.toml declares. The outcome for a scan author is the same either way: naming it in packages: overrides it.

packages:
  - inspect-scout==<version>

The two overrides are independent — overriding inspect-scout leaves inspect-ai pinned. Scout only declares a lower bound on inspect-ai, so releasing both would float inspect-ai to whatever is newest rather than to a pairing anyone has tested. If the pinned inspect-ai doesn't satisfy the Scout you asked for, resolution fails and you should name a compatible inspect-ai too:

packages:
  - inspect-scout==<version>
  - inspect-ai==<version scout accepts>

See Running Scans for the scan packages: field itself.

The override is matched by package name, so the entry has to name the package. A bare URL — the form shown under Additional Packages — won't be recognised, because it names no package until it is built:

packages:
  # Recognised: Hawk drops its inspect-ai pin.
  - inspect-ai @ git+https://github.com/my-org/inspect_ai_fork.git@my-fix

  # Not recognised: Hawk can't tell what this provides, so it keeps its pin
  # and the job fails with "no version of inspect-ai==<pinned version>".
  - git+https://github.com/my-org/inspect_ai_fork.git@my-fix

The same applies to any dependency that demands a newer inspect-ai than the pinned one — a task or scanner package declaring inspect-ai>=<something newer> makes the venv unresolvable and the job fails at install rather than quietly upgrading. Name a compatible inspect-ai in packages: to take control.

[Experimental] Custom Runner Images¶

You can use your own Docker image for the runner instead of the default:

runner:
  image: "<ecr-url>/custom-runners:my-image-v1"

Or via the CLI: hawk eval-set config.yaml --image <image-uri>

Images must have an explicit tag (:v1) or digest (@sha256:...). Tagless and :latest are rejected.

The image must be multi-arch or match the deployment's hawk:cpuArchitecture setting. A custom amd64-only runner requires an amd64 deployment.

Each Hawk deployment includes a custom-runners ECR repo with immutable tags. Get its URL with pulumi stack output custom_runners_ecr_url. Public images from any registry also work.

Look at the dockerfile in infra/runner-image/ to what a valid image looks like.

Sandbox Networking and Hardening¶

Sandbox network access is declared in your task's compose file or Kubernetes values.yaml, not in the eval-set config. Sandboxes are deny-by-default: no internet unless the task opts in, and network_mode: none isolates a service from its siblings as well.

See Sandbox Network Isolation for the full rules.

For evals that run code you expect to be hostile (cyber and offensive-security tasks, misuse probes), set isolation: strict and see Hardening High-Risk Evals.

GPU Tasks¶

GPU node pools are tainted (nvidia.com/gpu: NoSchedule) so that only workloads that need a GPU land on them. Any sandbox service that requests nvidia.com/gpu in its resource limits or requests automatically gets a toleration for that taint injected — task authors don't need to declare one. If a service already declares its own toleration that tolerates the taint (for example operator: Exists on the nvidia.com/gpu key), Hawk leaves the service's tolerations untouched. Operators can change what gets injected with the hawk:gpuNodeTolerations stack config option.

Monitoring¶

Logs¶

hawk logs                 # last 100 log lines for current job
hawk logs -f              # follow logs in real-time
hawk logs -n 50           # last 50 lines
hawk logs JOB_ID -f       # follow a specific job

Status¶

hawk status               # JSON report: pod state, logs, metrics, performance statistics
hawk status --hours 48    # include 48 hours of log data

The performance block (queue wait, working vs. waiting sample time, model-call and retry/error counts) is aggregated from the warehouse.

Note

Samples reach the warehouse only once their eval log is imported, so the performance statistics cover finished samples — a mid-run report excludes the samples still in flight. Use hawk watch for live per-sample progress; it reads the runner's live sample buffer instead. For scan jobs performance is null.

Web Viewer¶

hawk web                  # open current eval set in browser
hawk web EVAL_SET_ID      # open a specific eval set
hawk view-sample UUID     # open a specific sample

Listing and Inspecting Results¶

hawk list eval-sets                     # list all eval sets
hawk list evals [EVAL_SET_ID]           # list evals in an eval set
hawk list samples [EVAL_SET_ID]         # list samples
hawk transcript UUID                    # download a sample transcript (markdown)
hawk transcript UUID --raw              # download as raw JSON
hawk transcripts [EVAL_SET_ID]          # download all transcripts

Running Locally¶

Run evals on your own machine instead of the cluster. Useful for debugging.

hawk local eval-set examples/simple.eval-set.yaml

This creates a fresh virtualenv in a temp directory, installs dependencies, and runs the evaluation the same way the cluster would.

Providing model API keys¶

Unlike cluster runs, hawk local does not route model calls through Hawk's managed LLM proxy (Middleman). Inspect calls each provider's API directly, so you must supply the provider API keys yourself — otherwise the run fails with authentication errors from the provider.

Set the environment variables for the providers your models use, for example:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
hawk local eval-set examples/simple.eval-set.yaml

Or keep them in a file and load it with --secrets-file:

# .env
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

hawk local eval-set examples/simple.eval-set.yaml --secrets-file .env

You can also forward individual variables from your current shell with --secret NAME (see Secrets and API Keys above). Generate keys from your provider's dashboard (e.g. platform.openai.com, console.anthropic.com).

Note

To route through a managed gateway instead of using your own keys, see Using an AI Gateway below.

Debugging with `--direct`¶

Use --direct to skip the virtualenv and run in your current Python environment:

hawk local eval-set examples/simple.eval-set.yaml --direct

This lets you set breakpoints in your IDE and debug from the start. Note that --direct installs dependencies into your current environment via uv pip install, but model-provider packages (openai, anthropic, etc.) must already be present in the environment hawk was installed into. If they're missing, add them when installing hawk:

uv tool install --reinstall-package hawk "hawk[cli,runner]" --with openai

Note

Tasks from inspect_evals require openai even with a different model provider — inspect_evals imports it at load time, and missing it makes the task registry silently fail with LookupError.

Using an AI Gateway¶

Route model calls through a managed AI gateway:

export HAWK_AI_GATEWAY_URL=https://your-gateway.example.com
hawk login
hawk local eval-set examples/simple.eval-set.yaml

Sample Editing¶

Batch edit sample scores or invalidate samples:

hawk edit-samples edits.json

Accepts JSON arrays or JSONL:

[
  {"sample_uuid": "...", "details": {"type": "score_edit", ...}},
  {"sample_uuid": "...", "details": {"type": "invalidate_sample", ...}}
]

Importing Eval Files¶

Upload locally-produced .eval files into Hawk's warehouse so they appear alongside natively-run eval sets:

hawk import path/to/file.eval               # single file
hawk import path/to/dir/                     # directory of .eval files
hawk import path/to/dir/ --name my-import    # friendly name in the eval_set_id

Imported eval sets get IDs prefixed imported- and have metadata.imported = true set. They appear in the warehouse, viewer, and hawk download identically to natively-run eval sets.

Importing Scans¶

Upload one or more locally-produced Scout scans into the warehouse so their scanner results appear alongside natively-run scans:

hawk scan import path/to/scan_id=.../               # a single scan results directory
hawk scan import path/to/run/                        # a folder of scan directories
hawk scan import path/to/run/ --name my-import

PATH is either a single Scout scan results directory (the scan_id=... directory Scout writes) or a folder containing several such directories. Each scan directory must contain a _scan.json spec and at least one per-scanner .parquet file (_summary.json is uploaded too if present). All the scans in one import land under a single fresh, imported--prefixed scan run, each with a freshly-generated scan id, so they never collide with existing warehouse scans. The scanned transcripts must already be in the warehouse: Hawk derives the models that gate access (the scanner models plus the models of the scanned transcripts' source eval sets, matching what a native hawk scan run over those eval sets would require) and refuses to import if any scanned transcript can't be resolved, or if you lack permission for those models.

Human-in-the-Loop Approvals¶

For sensitive workloads you can require a human decision before the agent's tool calls execute. Two config pieces work together:

approval — a chain of Inspect approvers, matched top-down against each tool call. Approvers are resolved by registry name: Inspect's built-ins (human, auto) work out of the box, and packages listed in packages can register their own.
acp_server — a loopback TCP port (1–65535) that starts Inspect's ACP server inside the eval, so a human approver parks the tool call until someone attaches and answers, instead of the eval crashing or hanging on stdin.

A minimal end-to-end test config (a one-sample game where every tool call needs approval):

acp_server: 4444
approval_timeout_minutes: 60

approval:
  approvers:
    - name: human
      tools: ["*"]

tasks:
  - package: inspect-test-utils==1.5.0
    name: inspect_test_utils
    items:
      - name: guess_number_keep_guessing
        args: { sample_count: 1, local: true, target: "3" }

models:
  - package: openai
    name: openai
    items:
      - name: gpt-4o-mini

limit: 1

The flow:

hawk eval-set approval-test.yaml   # submit; the first tool call parks
hawk watch                         # pending_interactions goes to 1 (status stays "running")
hawk acp                           # bridge through the relay, open the `inspect acp` TUI

hawk acp defaults to the last job you launched from this machine; pass an eval-set id to answer a different run. It shells out to inspect acp, so you need an inspect on your PATH from an environment that has ACP support — the inspect-ai that ships with hawk[cli] does. If that binary is missing or broken, hawk acp says so and you can bridge without launching a client: hawk acp --no-launch, then point your own inspect acp --server <address> at the printed address.

In the TUI, pick the session and answer each request: approve executes the call, reject refuses just that call (the agent continues and may try something else), and terminate ends the sample gracefully — it is scored with the work done so far and recorded with a SampleLimitEvent, not an error.

Chain semantics: each tool call is handled by the first entry whose tools pattern matches; an approver that returns escalate hands the call to the next matching entry, and if every approver escalates the call is rejected. Narrow the tools patterns (they are prefix matches, e.g. "bash*") so humans only see genuinely risky calls — that is the safest way to cut approval volume, because a call that matches no policy never needed a decision in the first place.

An auto-approving approver is a noise filter, not a safety boundary

Any approver that returns approve decides that a tool call runs with nobody having seen it. Judging a shell command safe by inspecting it is unreliable: an allowlist keyed on program names has been shown to pass commands that write and then execute a file, and narrowing the character filter does not fix it. Put human (or a model-based monitor) where the safety decision actually matters, and treat anything in front of it as a convenience that only ever handles calls you would have approved anyway.

Local runs honor both settings too: hawk local eval-set starts the ACP server on the configured loopback port on your own machine, so an ACP client connects to it directly and no relay is involved.

approval_timeout_minutes bounds how long a parked approval waits before it is automatically denied — the default is one week; set it to null to wait indefinitely. It requires acp_server. A timed-out approval rejects that one tool call; the sample keeps running.

hawk acp requires the deployment's attach relay (relayEnabled, on by default). While approvals are parked the sample holds its sandbox, so pair long timeouts with monitoring on pending_interactions.

Budget for the volume: a model can emit several tool calls in one turn, and each one parks separately — a single turn has been observed producing ten simultaneous approvals. Narrow tools patterns accordingly, and expect a burst rather than a steady trickle.

Running Human Evaluations¶

For evaluations driven by a human instead of an LLM agent, Hawk provisions a sandbox and exposes it via SSH through the shared jumphost. config.yaml can be any eval-set YAML — the server swaps in the configured human-agent solver, clamps epochs=1 / limit=1, and defaults runner.cleanup to false so the sandbox sticks around between SSH sessions. To set args on the installed agent (e.g., user, record_session), add a human_eval.agent_args block to your config; it's shallow-merged onto the agent. Pass --no-rewrite to keep your config's agents/solvers block untouched (the SSH key is still injected on every solver).

# 1. Register the human and their SSH public key (one-time)
hawk human register --name jane --ssh-key "ssh-ed25519 AAAA..."

# 2. Start the eval — provisions a sandbox and registers jane's key on the jumphost
hawk human eval start config.yaml --human jane

# 3. Get a ready-to-paste SSH command for the sandbox
hawk human eval ssh-command          # uses the last-started eval-set
# — or —
hawk human eval ssh-command <eval-set-id>

ssh-command polls the eval logs for the agent's connection line and prints ssh -J ssh-user@<jumphost> <user>@<sandbox-ip> -p <port>. Load your private key into ssh-agent first (ssh-add /path/to/key) — the -J ProxyJump uses the agent for both the jumphost and sandbox hops. Use --timeout SECONDS to bound how long it waits for the sandbox to come up (default 600).

After hawk delete <eval-set-id> (or eval completion), the SSH key is removed from the jumphost and ssh-command refuses subsequent invocations.

Stopping and Deleting¶

hawk stop tells the running eval to finish gracefully: active samples are scored with whatever work they've done so far, results are written to S3, and the job exits on its own. Use this to end an eval early but keep the partial results. It also ends a runner kept alive after completion by runner.cleanup: false (e.g. human baselines).

hawk delete kills the job immediately and tears down its Kubernetes resources. Use this when you don't care about partial results and just want the job gone. Logs already written to S3 are kept. It takes either an eval-set ID or a scan-run ID, and confirms the target before tearing anything down. Scripts and other non-interactive callers must pass -y; without a terminal to prompt at, the command refuses rather than reading an answer from redirected stdin. -y also requires the ID to be given explicitly — with no prompt there is nothing to show which job the default resolved to, and that default moves whenever any command is given a job ID, including read-only ones like hawk logs.

hawk stop                  # gracefully stop current eval set (score partial work)
hawk stop --error          # mark samples as errors (will retry if retries are configured)
hawk stop --sample UUID    # stop a single sample

hawk delete                # tear down the current job's Kubernetes resources (logs are kept)
hawk delete JOB_ID         # delete a specific eval set's or scan run's resources
hawk delete JOB_ID -y      # skip the confirmation prompt (required in scripts;
                           # -y always needs the ID spelled out)

Running Evaluations¶

The Eval Set Config¶

Adding Agents¶

Eval Parameters¶

Secrets and API Keys¶

AWS Secrets Manager¶

Additional Packages¶

Overriding inspect-ai¶

[Experimental] Custom Runner Images¶

Sandbox Networking and Hardening¶

GPU Tasks¶

Monitoring¶

Logs¶

Status¶

Web Viewer¶

Listing and Inspecting Results¶

Running Locally¶

Providing model API keys¶

Debugging with --direct¶

Using an AI Gateway¶

Sample Editing¶

Importing Eval Files¶

Importing Scans¶

Human-in-the-Loop Approvals¶

Running Human Evaluations¶

Stopping and Deleting¶

Debugging with `--direct`¶