Example Configurations

The commands on this page use files from a cloned Hawk repository. Run them from the repository root. If you installed only the CLI, save a displayed YAML block to the filename in its title and pass that local path instead.

Simple Eval Set

A minimal evaluation that runs a small built-for-testing task (the model binary-searches a hidden number). The task installs from public PyPI, runs in-process (no sandbox pod), and is sized to fit a fresh AWS account's default quotas:

simple.eval-set.yaml

tasks:
  - package: inspect-test-utils==1.5.0
    name: inspect_test_utils
    items:
      - name: guess_number
        args:
          sample_count: 1
          local: true # run in-process — no sandbox pod needed

# Set the model to match a provider key you configured (Quick Start step 6).
# Middleman autodiscovers standard models for every configured provider, e.g.:
# openai/gpt-4o-mini, anthropic/claude-sonnet-4-6, google/gemini-2.5-flash
models:
  - package: openai
    name: openai
    items:
      - name: gpt-4o-mini

runner:
  cpu: "1" # fits a fresh account's default 8-vCPU EC2 quota
  memory: 2Gi

Submit it:

hawk eval-set hawk/examples/simple.eval-set.yaml

Eval Set with Secrets

Declares required secrets so Hawk catches missing credentials before the job starts:

simple-with-secrets.eval-set.yaml

tasks:
  - package: "git+https://github.com/UKGovernmentBEIS/inspect_evals@dac86bcfdc090f78ce38160cef5d5febf0fb3670"
    name: inspect_evals
    items:
      - name: mbpp
      - name: class_eval

models:
  - package: openai
    name: openai
    items:
      - name: gpt-4o-mini

runner:
  secrets:
    - name: OPENAI_API_KEY
      description: "OpenAI API key for model access"
    - name: HF_TOKEN
      description: "HuggingFace token for dataset access"

limit: 1

Submit with secrets:

hawk eval-set hawk/examples/simple-with-secrets.eval-set.yaml \
  --secret OPENAI_API_KEY --secret HF_TOKEN

Docker-in-Sandbox

Demonstrates running Docker containers inside sandbox pods:

docker-in-sandbox.eval-set.yaml

name: docker_in_sandbox_demo
tasks:
  - package: "git+https://github.com/METR/inspect-action.git@docker-in-sandbox-demo#subdirectory=demo/docker_in_sandbox_task"
    name: docker_in_sandbox_task
    items:
      - name: docker_in_sandbox_hello
        args:
          sample_count: 1
models:
  - package: "inspect-test-utils==1.5.0"
    name: hardcoded
    items:
      - name: hardcoded
        args:
          tool_calls:
            - tool_name: bash
              tool_args:
                cmd: "dockerd-entrypoint.sh >/tmp/dockerd.log 2>&1 & ..."
          answer: "hello"

Simple Scan

Runs Scout scanners over selected transcripts from previous evaluations:

simple.scan.yaml

scanners:
  - package: "git+https://github.com/METR/inspect-agents@metr_scanners/v0.1.0#subdirectory=packages/scanners"
    name: metr_scanners
    items:
      - name: reward_hacking_scanner
      - name: sandbagging_scanner
      - name: broken_env_scanner
        # Optional: names the results file (defaults to `name`). Letters, digits, `.`, `_`, `-` only.
        key: broken_env

models:
  - package: openai
    name: openai
    items:
      - name: gpt-4o-mini

transcripts:
  filter:
    where:
      # Sample UUIDs; list an eval set's with: hawk list samples EVAL_SET_ID
      - transcript_id: [SAMPLE_UUID_1, SAMPLE_UUID_2]
  # `hawk scan run` looks up the eval sets holding these transcripts.
  # `hawk local scan` can't, so uncomment this for local runs:
  # sources:
  #   - eval_set_id: YOUR_EVAL_SET_ID  # find yours with: hawk list eval-sets

Replace the SAMPLE_UUID_* placeholders with the transcripts to scan, then submit it:

hawk scan run hawk/examples/simple.scan.yaml