Skip to content

Running human evaluations on Hawk

A human evaluation runs the same Inspect task an AI agent would run, but a person solves it instead of a model. Hawk provisions the normal task sandbox, gives you SSH access to it through a shared jumphost, and records your session, timing, and final answer as a regular sample. So human baselines land in S3 and the data warehouse alongside agent runs and can be compared in the same tools.

This guide is for researchers running baselines themselves (authoring configs, managing the human registry, monitoring, and cleanup). For general eval-set concepts (tasks, agents, secrets, the config schema), see Running Evaluations.

Assigning a task to an external baseliner who doesn't have permission to run hawk commands on your infra? Hand them Completing a task baseline instead — it's written for someone who only needs to send an SSH key and connect, with none of the operator steps. Your side of that flow is: collect their public key, hawk human register it (step 1), hawk human eval start (step 3), then send them the hawk human eval ssh-command output (step 4).

How it works

hawk human eval start ─┬─► Hawk API ─► provisions the task sandbox pod
                       │                 (runs the human_agent in it)
                       └─────────────► registers your SSH public key on
                                        the shared jumphost

hawk human eval ssh-command ─► reads the sandbox's SSH details from the run logs

you ──ssh -J jumphost──► sandbox ──`task submit`──► sample scored ─► S3 + warehouse
                                                          └─► your key removed
                                                              from the jumphost

The pieces:

  • Human registry: a one-time record of your name + SSH public key (hawk human ...). Hawk injects the registered key into the agent automatically, so you never paste it into a config.
  • human_agent: the in-sandbox program (from metr_agents) that starts an SSH server, drops you into the task's working directory, shows the task instructions, and gives you the task command for submitting.
  • Jumphost: a shared bastion. Sandboxes aren't reachable directly; you connect through it with ssh -J. Your key is added when the eval starts and removed when it finishes.

Prerequisites

  • A reasonably recent hawk CLI (human-eval support is on main). Examples below use uv run hawk from a clone; a globally installed hawk works too.
  • An SSH client and ssh-keygen (standard on macOS/Linux).
  • HAWK_API_URL pointed at your environment, and an authenticated session:
hawk login            # or: hawk login --no-browser  (headless / SSH boxes)

--no-browser prints a URL you open in any browser. Depending on what your provider supports, you'll either approve a short user code (nothing to copy back) or paste the redirect URL your browser lands on back into the CLI. Let the command run to completion so the token is stored.

1. Register yourself (one-time)

Create an SSH key pair (or reuse an existing one):

ssh-keygen -t ed25519 -f ~/.ssh/id_hawk -C "me@metr.org"

Register the public key under a name:

hawk human register --name me-metr-org --ssh-key "$(cat ~/.ssh/id_hawk.pub)"

Notes:

  • Name rules: lowercase letters, digits, and hyphens; 1–63 chars; must start and end with a letter or digit (Kubernetes label rules). No dots or @, so an email address won't work as a name. The name is independent of the comment in the key.
  • Key types: ssh-ed25519, ssh-rsa, the ecdsa-sha2-nistp{256,384,521} types, and the sk-*@openssh.com FIDO variants.
  • Registering a name that already exists is rejected. Rotate your key with update instead. Only the account that registered the name (or a Hawk admin) can update or delete it. Registry rows created before ownership existed have created_by NULL and stay admin-only for update/delete; teams can still start evals under that name. There is no claim API. An admin can delete the row and re-register it (the new registrant becomes owner; use --force if it has assignment history) or set created_by in SQL to the intended owner's JWT sub.

Managing the registry:

hawk human list                                   # name, key fingerprint, created_at
hawk human update me-metr-org --ssh-key "$(cat ~/.ssh/id_hawk_new.pub)"   # rotate your key (owner or admin)
hawk human delete me-metr-org                     # remove (owner or admin; use --force if it has past assignments)

2. Write an eval-set config

Any existing eval-set YAML works as a human eval — the server swaps in the configured human_agent for you, so the simplest config is just a tasks block. Example (mbpp_human_eval.yaml):

name: mbpp_human_eval
tasks:
  - package: "git+https://github.com/UKGovernmentBEIS/inspect_evals@dac86bcfdc090f78ce38160cef5d5febf0fb3670"
    name: inspect_evals
    items:
      - name: mbpp
        sample_ids:
          - "11"        # omit to run the whole task

What the server does to your config when you start a human eval:

  • Replaces agents/solvers with the configured human_agent. You don't need to write either block (and any you provide is overwritten — see the --no-rewrite escape hatch below).
  • Forces epochs=1 and limit=1 so you do one sample at a time.
  • Injects your registered SSH key as args.public_key on the agent.
  • Defaults runner.cleanup to false so the sandbox sticks around between SSH sessions; set it explicitly to override.

So tasks is the only required block. sample_ids picks specific samples; drop it to run them all. Each sample is a separate sandbox you'll connect to in turn. Other eval-set options apply too — --secret/--secrets-file for credentials, and runner: overrides such as memory: 2Gi for heavier tasks. See Running Evaluations.

Network access comes from the task, not from this config. The sandbox you SSH into gets exactly the network the task's sandbox definition (its Docker Compose file or Kubernetes values.yaml) grants, and that is deny-by-default: no internet unless the task opts in with network_mode: bridge, a single non-internal bridge network every service joins, or an allowlist. A human eval only adds the SSH ingress on port 2222; nothing in the human_eval block or any other Hawk-level setting overrides the task's policy. Task args are the one lever in your YAML, and only when the task exposes one. If you need internet, pick a task that grants it, or one whose args let you turn it on. See Sandbox Network Isolation.

The default human_agent runs a small Python task script inside the sandbox, so the task image must have python3.

Setting agent args without --no-rewrite

For most arg tweaks (a non-default user, record_session=false, etc.), add a human_eval.agent_args block to your config — the rewrite shallow- merges it onto the operator-configured default agent. Example:

human_eval:
  agent_args:
    user: root
    record_session: false

The same config remains usable for hawk eval-set (the human_eval block is ignored on non-human runs).

Advanced: ship your own agents block (--no-rewrite)

If you need a forked or custom human-agent solver entirely, write the agents block yourself and pass --no-rewrite in the next step. The server then leaves your config alone, only injecting your registered SSH key as args.public_key on every agent/solver. Useful args on the stock metr_agents human_agent:

  • user — the Linux user you log in as (default agent). Use root for images that only have root, like mbpp.
  • record_session — capture a script(1) recording of your terminal (default true).
  • workdir — directory your shell starts in (defaults to the sandbox's working directory, i.e. where the agent's tools would run).
  • Don't set public_key — Hawk injects your registered key for you. If you do set it, Hawk overwrites it with the registry value.

3. Start the eval

hawk human eval start mbpp_human_eval.yaml --human me-metr-org

Any principal who can create an eval may pass --human for a registered name (a team / shared baseliner). This injects the registered key, creates the eval set, and registers the key on the jumphost. It prints the eval-set ID plus log-viewer and Datadog links, and remembers the ID so later commands default to it:

Eval set ID: me-metr-or-mbpp-aBcD1234
Human: me-metr-org
See your eval set log: https://...
Monitor your eval set: https://...

Use --priority low|medium|high on hawk human eval start to choose the job's priority. The default is medium.

If the deployment enables its outstanding-Job submission guard, this launch path is guarded too. --force skips the guard and its count for one coordinated submission. Authentication, config validation, and native admission still apply.

4. Connect to the sandbox

Fetch a ready-to-paste SSH command. It polls the run logs until the sandbox is up and the agent prints its connection details. If the config has several tasks or samples, each gets its own sandbox and ssh-command prints the one that reported most recently, so keep a baseline config to one task unless you mean to work through several:

hawk human eval ssh-command            # defaults to the last-started eval set
# hawk human eval ssh-command <eval-set-id>          # or be explicit
# hawk human eval ssh-command --timeout 900          # wait longer (default 600s)

It prints something like:

# Load your private key into the SSH agent first:
#   ssh-add /path/to/your/private-key
ssh -J ssh-user@jumphost.example.com root@10.0.0.5 -p 2222

Load your key into the agent (the -J ProxyJump uses it for both hops), then paste the command:

ssh-add ~/.ssh/id_hawk
ssh -J ssh-user@jumphost.example.com root@10.0.0.5 -p 2222

You can disconnect and re-run ssh-command to reconnect at any time until you submit. For long tasks, run tmux (if present) inside the sandbox so a dropped connection doesn't lose your shell.

Host key warnings

Sandbox IPs are reused across runs, so SSH may refuse to connect with REMOTE HOST IDENTIFICATION HAS CHANGED. The warning names the offending entry (e.g. 10.0.0.5:2222); remove it and reconnect:

ssh-keygen -R '10.0.0.5:2222'

Only do this when the change is expected (a fresh sandbox or jumphost was provisioned).

5. Do the task

On first login the agent shows the task instructions (also saved to ~/instructions.txt), starts the timing clock, and begins recording your session. You're logged in as the configured user, and your shell starts in the task's working directory — the same place an agent's tools would run.

Complete the task in the shell, then use the task command:

Command What it does
task instructions Reprint the task instructions and the command list.
task status Show whether the clock is running/paused and elapsed time.
task pause / task resume Pause the clock for a break; resume to keep working.
task note [message] Record a note in the transcript (opens $EDITOR if no message).
task submit [answer] Submit your final answer and end the task.
task quit End the task without submitting an answer.

Notes:

  • task submit validates your answer first. Some tasks require a non-empty answer or one matching a specific format; if it's rejected, re-read task instructions. Tasks with no required answer accept a bare task submit.
  • submit and quit ask for confirmation and then disconnect you permanently — once submitted you can't reconnect, so don't run them until you're done.
  • The clock must be running to submit (task resume if you paused).

When you submit, the sample is scored, results are written to S3 and the warehouse, the sandbox is torn down, and your key is removed from the jumphost. If your config has multiple samples/epochs, re-run ssh-command to connect to the next sandbox.

Monitoring a run

hawk status <eval-set-id>      # JSON: pod state, logs, metrics
hawk logs <eval-set-id> -f     # follow the run logs
hawk web <eval-set-id>         # open the eval set in the browser viewer

(start also prints the log-viewer and Datadog links.)

Finishing, aborting, and cleanup

  • Normal finish: task submit (or task quit) inside the sandbox.
  • Abort from outside the sandbox:
  • hawk stop <eval-set-id> — stop gracefully; in-progress work is scored.
  • hawk delete <eval-set-id> — tear down the Kubernetes resources immediately (logs are kept).
  • Completion, stop, and delete all remove your key from the jumphost. After that, ssh-command refuses (already completed … start a new human-eval) — start a fresh eval to do the task again.

Troubleshooting

  • ssh-command times out — the sandbox is still starting (image pulls can be slow). Raise --timeout and watch hawk logs <id> -f for the SSH connection: line.
  • ssh-command refuses, or prints with a warning — before printing anything, ssh-command confirms the sandbox pod's Kubernetes status. There are three outcomes:
  • Live and ready — prints the command normally (after the SSH connectivity check described below, unless --skip-verify was passed).
  • Live status unknown (e.g. a transient status request failure, a Kubernetes API error, or the eval-set is brand new and hasn't produced any warehouse data yet) — prints the command anyway with a "could not confirm the sandbox pod's live status" warning, since there's no evidence the pod is unhealthy.
  • Confirmed absent or not ready (no live pod, not Running, or a container isn't ready) — refuses outright, naming the observed phase/container state. Run hawk watch <eval-set-id> to check current status and retry once it settles.

If the logged endpoint identifies a live sandbox pod, the command uses that pod for the liveness check. Otherwise, it keeps the logged IP and warns rather than connecting you to a different task. - ssh-command prints the command with a "sandbox endpoint was not verified" warning — a separate, network-level check (distinct from the Kubernetes pod-liveness check above) couldn't confirm the SSH endpoint itself is reachable, and the warning says which hop got in the way. If it never reached the jumphost, that's usually your own VPN / Tailnet or an unloaded key. If it reached the jumphost but couldn't confirm the sandbox, the sandbox may still be starting or its SSH server may not be running — the check only retries for about 20s, so another ssh-command is worth a try. Hawk provides the port-2222 ingress itself, so this is never a network policy you have to grant (see Sandbox Network Isolation). --skip-verify skips only this SSH connectivity probe — the Kubernetes pod-liveness check above always runs and can still refuse. - Permission denied (publickey) — your key isn't in the agent (ssh-add ~/.ssh/id_hawk), the loaded key doesn't match the one you registered, or --human named a different person. Check hawk human list. - No internet from inside the sandbox (apt update or curl fail with Temporary failure resolving ...) — expected unless the task's sandbox definition grants it; see the note under Write an eval-set config. If the config lists several tasks, check which sandbox ssh-command gave you: it picks the most recently reported one, which may be a task without internet. - task: command not found — the task image lacks python3, or your shell didn't source the agent's .bashrc. Re-login, or source ~/.bashrc. - REMOTE HOST IDENTIFICATION HAS CHANGED — stale host key; see Host key warnings. - First-connect "authenticity of host … can't be established" — expected for a new jumphost/sandbox; accept to continue. - start fails registering the key on the jump host (502) — Hawk rolls the eval back automatically; just retry. If the error says manual cleanup is needed, run hawk delete <eval-set-id>.

Command reference

# Registry
hawk human register --name <name> --ssh-key "<pubkey>"
hawk human list
hawk human update <name> --ssh-key "<pubkey>"   # owner or admin
hawk human delete <name> [--force]              # owner or admin

# Run a human eval
hawk human eval start <config.yaml> --human <name> [--secret NAME] [--secrets-file FILE] [--force]
hawk human eval ssh-command [<eval-set-id>] [--timeout SECONDS] [--skip-verify]

# Inside the sandbox
task instructions | status | pause | resume | note [msg] | submit [answer] | quit

# Monitor / clean up
hawk status <eval-set-id>
hawk logs <eval-set-id> -f
hawk web <eval-set-id>
hawk stop <eval-set-id>
hawk delete <eval-set-id>