Skip to content

Running Scout Scans

Scans run Inspect Scout scanners over transcripts from previous evaluations.

Scan Config

```yaml scanners: - package: git+https://github.com/METR/inspect-agents name: metr_scanners items: - name: reward_hacking_scanner - name: sandbagging_scanner - name: broken_env_scanner

models: - package: openai name: openai items: - name: gpt-5

transcripts: sources: - eval_set_id: inspect-eval-set-abc123 filter: where: - eval_status: success limit: 10 shuffle: true ```

Submitting a Scan

bash hawk scan run config.yaml

Pass secrets the same way as evaluations:

bash hawk scan run config.yaml --secret MY_API_KEY --secrets-file .env

Transcript Filtering

Filter which transcripts to scan:

yaml transcripts: filter: where: - eval_status: success # Only successful runs - score: {gt: 0.5} # Score above threshold - model: {like: "gpt-4%"} # Pattern matching - metadata.agent.name: react # Nested field access limit: 100 shuffle: true

Available Operators

  • Equality: field: value or field: [val1, val2] (IN list)
  • Comparison: {gt: 0.5}, {ge: 0.5}, {lt: 1.0}, {le: 1.0}, {between: [0.5, 1.0]}
  • Pattern matching: {like: "pattern"}, {ilike: "PATTERN"} (case-insensitive)
  • Logical: {not: condition}, {or: [cond1, cond2]}
  • Null checks: field: null

Per-scanner filters can override the global filter via scanners[].items[].filter.

Resuming Scans

Scan configs are saved to S3. Resume an interrupted scan:

bash hawk scan resume # resume the last scan hawk scan resume SCAN_RUN_ID # resume a specific scan

Secrets must be re-provided via --secret or --secrets-file when resuming.

Running Scans Locally

bash hawk local scan examples/simple.scan.yaml hawk local scan config.yaml --secrets-file .env --secret MY_API_KEY