Auditing model-group labels

A run's model groups decide who can read it. They are stored in several places:

• the folder policy (.models.json), which guards the raw logs;
• eval.model_groups and scan.model_groups, which guard the warehouse rows;
• sample_model, which feeds the warehouse groups;
• the restricted floor of a run launched with restricted credentials, saved in the warehouse and copied into the folder policy.

Each copy is written from what an import could see at the time, and a model's group can change later. The audit (python -m hawk.tools.audit_model_groups) re-reads every .eval log for the models its samples called and reports any copy that lacks a group the logs' models need today, or that the warehouse rows store. It also asks the API's own folder permission check whether a reader missing such a group gets in. Scan folders are found from their warehouse rows, so a scan folder with no rows isn't audited.

It is read-only. It runs as the warehouse login inspect-audit, which reads every row (it holds rls_bypass) of the tables it audits (eval, sample, sample_model, scan) and writes nothing: those SELECT grants, plus EXECUTE on the five functions the audit calls. Those rows include sample inputs and outputs, as do the logs its job reads. Of scanner_result and code_policy it reads only the columns that link scans to their sources. Like every rls_bypass member, it can also read job and Middleman's model registry.

Running it on a stack

Stacks that own their Batch compute have a Batch job for it, with its own Fargate Spot compute environment capped at 256 vCPUs, so it never takes the importer's compute. It does share the account's Fargate Spot vCPU quota with the importer. Dev stacks share stg's Batch compute, so they have none. Nothing starts it automatically: a person with admin access to the account submits it as an array job, one child per slice. The queue and the job definition are both named <env>-hawk-model-group-audit, and the job definition's bucket defaults to the stack's:

ENV=prd
RUN=$(date -u +%Y%m%d-%H%M%S)-$(openssl rand -hex 3)  # unique: a reused `out` skips saved runs
aws batch submit-job \
  --job-name "model-group-audit-$RUN" \
  --job-queue "$ENV-hawk-model-group-audit" \
  --job-definition "$ENV-hawk-model-group-audit" \
  --array-properties size=500 \
  --parameters "out=s3://<bucket>/audits/model-groups/$RUN/,shards=500"

Submit many more slices than run at once (64): slices are balanced by count, not size, so a queue of them absorbs the few that hold the largest eval sets.

Each child's log (CloudWatch, /aws/batch/<env>-hawk-model-group-audit) shows what it will audit, a status line every 30 seconds and when each phase ends (done and total, elapsed, time left, fails and warnings so far), and a summary by kind when it finishes. Follow the whole job with aws batch describe-jobs --jobs <job id> --query 'jobs[0].arrayProperties.statusSummary'.

out has no default: a submit without it fails, rather than resuming into an earlier run's results and skipping everything they cover.

Each child audits the runs that hash into its slice, concurrency eval sets and scan runs at a time:

• It parses an eval set's logs in parallel, one worker process per vCPU (--parse-workers). A log whose worker dies, most likely out of memory, is retried alone and is an unreadable-log if it dies again.
• It checkpoints each log of 32 MiB or more under partial/, so a reclaimed or timed-out slice resumes inside an eval set rather than parsing it all again.
• It saves one result file per run, audits/model-groups/<run>/evals/<id>.json or scans/<id>.json, with the run's findings and the ETag of every log they cover. An export can use the ETags to hand over only the log versions the audit read.
• When the slice finishes, it writes shards/<i>-of-<N>.json.

The job logs hold counts only, never run or model names: Hawk's and Inspect's own warnings are silenced in the job, and the findings go to the result files. Those name eval sets and models, some of them unreleased, so they stay in the bucket, which only admin-level roles read directly.

Reading the results

From hawk/ in a checkout, with credentials that can read the results. Where hawk:enableInspectDataBucketDeny is on, that means a role in hawk:inspectDataBucketBreakGlassPrincipalArns (BreakGlass in prd):

uv run python -m hawk.tools.audit_model_groups --report --shards 500 \
  --out s3://<bucket>/audits/model-groups/$RUN/ > findings.csv

This writes every finding as CSV to stdout, and a summary to the terminal: how many slices finished, how many eval sets and scan runs were audited, totals by kind, and the first 25 fails. --shards is the run's slice count, which must match the array size: the report names any of those slices that never wrote its completion record, since its runs went unchecked, and exits 1 on one, as on any fail finding. (A run of named eval sets or scans has no slices; report it with --partial instead, which a sliced run refuses. --partial can't tell whether every named run finished, so check that the run itself exited 0.) A fail means a readable copy is less restricted than the logs or the warehouse need, or that the audit couldn't tell:

Kind	Meaning
eval-groups	A readable eval lacks a group its logs' models need, or one its own retained floor (observed_model_groups) holds; for an eval with no log, one the warehouse itself derives (compute_eval_model_groups).
vouched-models	A readable folder's policy holds only the public group, yet it alone vouches for a model the registry lacks. With other groups this is a warning.
folder-groups	A folder policy lacks such a group, or one its warehouse rows store.
folder-floor	A folder's restricted floor lacks a group the warehouse saved for the run's executions, or, for a scan run, one its source eval sets' folder floors hold (a source with no .models.json fails too, since its floor can't be checked).
folder-access	The folder permission check admits a reader without a needed or floor group, or admits readers to unresolved scans.
unknown-models, folder-unknown-models	A readable eval or folder holds a model Middleman doesn't know and its folder's policy doesn't name alongside a group, so its group can't be required.
scan-groups	A readable scan lacks a group its models or sources need today.
unreadable-log, audit-error	A log or run couldn't be audited, including a named scan run with no warehouse scans and a scan whose location names no run folder.

Findings marked warn are informational: an unimported log, a sample whose sample_model rows lack a nested call's name, or a run no one can read. A folder-access warning says why the folder refused even a reader holding every group its runs need: it carries __unresolved__, lists no groups, or code access refuses it. Each of those is stricter, never looser.

A model the registry lacks but the folder's policy names, alongside at least one group, is one of Middleman's dynamic catalog models. The eval importer trusts it with every group that policy holds, so the audit requires all of them; the eval-groups fail lists the groups only such a model needs apart, and names it. Those are the folder's groups today: if the folder gained one after the eval's import (say, a later upload of a restricted model), the eval fails although its import was right, since no record says which groups the folder held then. Such a fail can't be told apart from a real gap, so add the group rather than dismiss it: that only hides the eval from more readers.

A warehouse eval with no log in S3, including every eval of an eval set whose folder is gone, is checked against the groups the warehouse itself derives for it (compute_eval_model_groups), and a no-log warning counts them per eval set.

A clean report means no copy is looser than the logs and the warehouse need. It doesn't mean a folder is public: before exporting one as public, check that its policy lists only the public group, that it has no restricted floor, and that its code access is public. The audit grants every code group, so it says nothing about code access.

The audit names a sample's models exactly as the importer does, so it shares the importer's blind spots. One to know: a model a nested call (inside a tool or subtask) sent under a rewritten wire name. The importer records a nested call's own model name but not that rewritten wire name, so the audit — like the importer that labelled the run — can't require its group. This bites only when such a wire name is a separately registered restricted model no other name in the run reveals; closing it is a converter change, tracked separately.

Imports in flight can produce mismatches that settle on their own. Before acting on a fail, re-run that eval set alone (--eval-set <id>, against the same --out), which always redoes it.

When a slice fails

Each child gets 4 vCPUs and 30 GiB. It parses up to four logs at once, splitting a log of 32 MiB or more across all four, and reads each sample for only the names of the models it called, not its whole content. A worker that runs out of memory anyway (a very large legacy nested transcript, say) is retried alone, so only a log too big to parse by itself fails. Batch retries a slice, up to 10 attempts, only when Fargate Spot reclaims it. A retried slice skips the runs it already saved and the logs it checkpointed, so it resumes rather than restarting. Any other failure ends the slice: most often running out of memory, or the 24-hour attempt limit on an enormous eval set. Submit that slice again, alone, with more resources or time. It also resumes:

RUN=<run>    # the failed run's, as in its `out`
SLICE=<i>    # from the slice --report names as missing
aws batch submit-job --job-name "model-group-audit-$RUN-$SLICE" \
  --job-queue "$ENV-hawk-model-group-audit" \
  --job-definition "$ENV-hawk-model-group-audit" \
  --timeout attemptDurationSeconds=172800 \
  --container-overrides "{\"command\": [\"--bucket\", \"<bucket>\",
    \"--out\", \"s3://<bucket>/audits/model-groups/$RUN/\", \"--shards\", \"500\", \"--shard\", \"$SLICE\"],
    \"resourceRequirements\": [{\"type\": \"VCPU\", \"value\": \"8\"}, {\"type\": \"MEMORY\", \"value\": \"61440\"}]}"

The command is overridden because Batch reserves AWS_BATCH_* variables: a single job can't set AWS_BATCH_JOB_ARRAY_INDEX itself.

On a first run in an environment, watch the warehouse's load: on a single-instance cluster the audit's reads go to the same instance that serves the API and the importers, and up to 64 children run at once. Submit a smaller array if it strains. concurrency defaults to 3; keep it at 3 or less, since each child's connection pool holds at most 10 connections. A run's checkpoints assume one image: resubmit its slices with the same job definition revision.

Running it elsewhere

Locally, or on any host that can reach the warehouse and the bucket, run the same module from hawk/ with DATABASE_URL set to a login that bypasses row-level security and sees code_policy's rows (it refuses to start otherwise). Save the results to a local folder:

DATABASE_URL=postgresql://... uv run python -m hawk.tools.audit_model_groups \
  --bucket <bucket> --out ./audit/ --eval-set <id>