Skip to content

Running Scout Scans

Inspect Scout is a companion tool to Inspect AI that runs automated scanners over transcripts from completed evaluations. Scanners analyze agent behavior in transcripts — for example, detecting reward hacking, unsafe tool use, or other safety-relevant patterns. Hawk integrates Scout so you can run scans at scale across your evaluation results, with the same infrastructure (Kubernetes pods, managed LLM proxy, results warehouse) used for evals.

Scan Config

scanners:
  - package: git+https://github.com/METR/inspect-agents@metr_scanners/v0.1.0#subdirectory=packages/scanners
    name: metr_scanners
    items:
      - name: reward_hacking_scanner
      - name: sandbagging_scanner
      - name: broken_env_scanner

models:
  - package: openai
    name: openai
    items:
      - name: gpt-5

transcripts:
  sources:
    - eval_set_id: inspect-eval-set-abc123
  filter:
    where:
      - eval_status: success
    limit: 10
    shuffle: true

Each scanner item may set an optional key (defaulting to its name) that names its results file. A key must contain only letters, digits, ., _ and - (and not be . or ..); other characters are rejected when the scan is submitted.

The submission priority applies to all scanner model calls through Middleman, including model roles. It overrides any x-middleman-priority header supplied by the scanner or model configuration.

Scan models, including model roles, default to cache_prompt: false. Scanners rarely reuse a prompt, so a scanner that doesn't ask for caching, such as metr_scanners, no longer pays for Anthropic prompt-cache writes it never reads. Set cache_prompt: true in a model's args.config to cache anyway. Scout's built-in llm_scanner asks for caching on every call with its default template, which overrides this setting either way. Scan requests that do cache keep Anthropic's 5-minute TTL (writes at 1.25x input) rather than the 1-hour TTL Middleman gives evals (2x); set cache_ttl: 1h in a model's args to keep the longer one, for example for a scan that asks several llm_scanner questions per transcript under heavy rate limiting. OpenAI's automatic caching is unaffected.

Submitting a Scan

hawk scan run config.yaml

Pass secrets the same way as evaluations:

hawk scan run config.yaml --secret MY_API_KEY --secrets-file .env

Admission priority

Use --priority low|medium|high to choose the priority for cluster admission and inference requests. The default is low for both launches and explicit resumes:

hawk scan run config.yaml --priority medium
hawk scan resume SCAN_RUN_ID --priority high

Each explicit resume uses its current option or the default. Automatic retries keep the job's priority.

These map to numeric priorities -100, 0, and 100 when scan runners are enrolled in Kueue. These values are Hawk's convention: their ordering matters, and the spacing has no special scheduling effect. Within the shared runner queue, this orders waiting Workloads from every user and Hawk deployment in the cluster. It grants no capacity entitlement or permission, does not preempt running work, and does not guarantee admission or startup.

Kueue uses BestEffortFIFO: a lower-priority scan that fits can be admitted while a higher-priority scan waits for unavailable capacity. Coordinate deliberate expensive scans with other cluster users. Kueue admission is enabled per deployment for all runners and standard-chart sandbox services. See the Kueue operator guide for configuration.

Submission guard

Deployments can enable an accidental-burst guard for outstanding runner Jobs. It is disabled by default and defaults to a threshold of 128 pending, queued, or running jobs for the deployment and submitting user. Jobs that completed successfully or failed do not count. New Jobs use a stable hashed subject label. Older Jobs without that label are counted conservatively through their legacy sanitized creator label in the expected deployment namespace; this can overcount if two legacy values collide, and it does not relabel those Jobs.

The guard covers both hawk scan run and hawk scan resume. A threshold refusal returns HTTP 429 with the observed count and limit, while a count failure returns HTTP 503. --force skips the guard and its count for one coordinated submission. Authentication, configuration validation, and native admission still apply. Its lock is local to one API process, so the guard is not an exact distributed quota.

Additional Packages

Scan configs support a top-level packages: field, same as eval sets — see Additional Packages for the general mechanism, and Overriding inspect-ai for how to override the pinned inspect-ai or inspect-scout a scan job installs.

Transcript Filtering

Filter which transcripts to scan:

transcripts:
  filter:
    where:
      - eval_status: success           # Only successful runs
      - score: {gt: 0.5}               # Score above threshold
      - model: {like: "gpt-4%"}        # Pattern matching
      - metadata.agent.name: react     # Nested field access
    limit: 100
    shuffle: true

Available Operators

  • Equality: field: value or field: [val1, val2] (IN list)
  • Comparison: {gt: 0.5}, {ge: 0.5}, {lt: 1.0}, {le: 1.0}, {between: [0.5, 1.0]}
  • Pattern matching: {like: "pattern"}, {ilike: "PATTERN"} (case-insensitive)
  • Logical: {not: condition}, {or: [cond1, cond2]}
  • Null checks: field: null

Per-scanner filters can override the global filter via scanners[].items[].filter.

Scanning specific transcripts

To scan particular transcripts, list their ids (sample uuids):

transcripts:
  filter:
    where:
      - transcript_id: [SAMPLE_UUID_1, SAMPLE_UUID_2]

When every scanner's filter (its own, or the global one) consists only of transcript_id conditions, as above:

  • the scan reads only the .eval files holding those transcripts instead of every log in the source eval sets, which makes scans over large eval sets much faster to start.
  • with hawk scan run, sources can be omitted: Hawk looks up the eval sets holding these transcripts in the warehouse. They can span at most 20 eval sets. hawk local scan still needs sources.

Hawk finds a transcript's file through the warehouse, so this depends on the transcripts being imported. If any id isn't in the warehouse yet, a scan with sources reads the whole source eval sets as before, and a scan without sources is refused, since Hawk can't tell which eval sets hold it.

Adding any other condition, such as source_uri or eval_status, keeps the usual behaviour: list sources, and the scan reads every log in them. Retried eval sets keep older copies of a transcript in earlier logs, and such conditions can select those copies.

Resuming Scans

Scan configs are saved to S3. Cluster scan jobs do not automatically retry failed runner pods; resume explicitly after investigating a failure:

hawk scan resume              # resume the last scan
hawk scan resume SCAN_RUN_ID  # resume a specific scan

Resume uses the saved configuration. If configuration or permission files are missing, submit the original configuration again with hawk scan run original.yaml to start a new scan. If the saved scan has no results yet, resume starts it normally.

Secrets must be re-provided via --secret or --secrets-file when resuming. Use --force only for the submission-guard exception described above.

Running Scans Locally

The first command uses the repository's example file. From the root of a cloned Hawk repository, replace its SAMPLE_UUID_* placeholders and uncomment its sources with the eval set holding them: a local scan can't look eval sets up in the warehouse. Otherwise, pass the path to your own scan config.

hawk local scan hawk/examples/simple.scan.yaml
hawk local scan config.yaml --secrets-file .env --secret MY_API_KEY --priority medium

--priority defaults to low and also works with --direct. It sets the inference priority when requests are routed through Middleman.