Skip to content

Running Scout Scans

Inspect Scout is a companion tool to Inspect AI that runs automated scanners over transcripts from completed evaluations. Scanners analyze agent behavior in transcripts — for example, detecting reward hacking, unsafe tool use, or other safety-relevant patterns. Hawk integrates Scout so you can run scans at scale across your evaluation results, with the same infrastructure (Kubernetes pods, managed LLM proxy, results warehouse) used for evals.

Scan Config

scanners:
  - package: git+https://github.com/METR/inspect-agents
    name: metr_scanners
    items:
      - name: reward_hacking_scanner
      - name: sandbagging_scanner
      - name: broken_env_scanner

models:
  - package: openai
    name: openai
    items:
      - name: gpt-5

transcripts:
  sources:
    - eval_set_id: inspect-eval-set-abc123
  filter:
    where:
      - eval_status: success
    limit: 10
    shuffle: true

Submitting a Scan

hawk scan run config.yaml

Pass secrets the same way as evaluations:

hawk scan run config.yaml --secret MY_API_KEY --secrets-file .env

Transcript Filtering

Filter which transcripts to scan:

transcripts:
  filter:
    where:
      - eval_status: success           # Only successful runs
      - score: {gt: 0.5}               # Score above threshold
      - model: {like: "gpt-4%"}        # Pattern matching
      - metadata.agent.name: react     # Nested field access
    limit: 100
    shuffle: true

Available Operators

  • Equality: field: value or field: [val1, val2] (IN list)
  • Comparison: {gt: 0.5}, {ge: 0.5}, {lt: 1.0}, {le: 1.0}, {between: [0.5, 1.0]}
  • Pattern matching: {like: "pattern"}, {ilike: "PATTERN"} (case-insensitive)
  • Logical: {not: condition}, {or: [cond1, cond2]}
  • Null checks: field: null

Per-scanner filters can override the global filter via scanners[].items[].filter.

Resuming Scans

Scan configs are saved to S3. Resume an interrupted scan:

hawk scan resume              # resume the last scan
hawk scan resume SCAN_RUN_ID  # resume a specific scan

Secrets must be re-provided via --secret or --secrets-file when resuming.

Running Scans Locally

hawk local scan examples/simple.scan.yaml
hawk local scan config.yaml --secrets-file .env --secret MY_API_KEY