Skip to content

Database

Each environment gets an Aurora PostgreSQL Serverless v2 cluster. Application connections use IAM authentication rather than long-lived database passwords, while RDS maintains an AWS-managed master-user secret that Pulumi uses through the Data API for provisioning. With the default hawk:dbMinCapacity: "0", the cluster scales to zero after a few minutes of inactivity to save costs; the first connection after a pause takes ~30 seconds to wake up.

Database Roles

These login roles are created automatically:

Role Purpose
inspect_admin Migrations (rds_superuser)
inspect API read/write
inspect_ro Read-only access
middleman LLM proxy model configuration reads and admin API writes
inspect-importer Import pipeline (eval-log and scan importers); bypasses RLS. Name configurable via hawk:warehouseSystemUser

Row-level security is managed through NOLOGIN group roles that the login roles are granted into: rls_bypass (system pipelines), rls_reader (RLS-filtered reads), and model_access_all. See Security: Access Control for how permissions flow.

Connecting

After logging Pulumi in to the correct S3 backend, name the target stack explicitly. Resolve both the database endpoint and AWS region from that stack, then generate an IAM auth token for your database role:

export AWS_PROFILE="<profile-for-the-target-account>"
STACK="<stack>"
ENDPOINT="$(pulumi stack output database_url_admin -s "$STACK" | sed -E 's#.*@([^:/]+).*#\1#')"
REGION="$(pulumi stack output region -s "$STACK")"
TOKEN="$(aws rds generate-db-auth-token \
  --hostname "$ENDPOINT" --port 5432 --region "$REGION" --username inspect)"
PGPASSWORD="$TOKEN" psql "host=$ENDPOINT dbname=inspect user=inspect sslmode=require"

Running Migrations

After logging Pulumi in to the correct S3 backend, get the database URL for an isolated development stack. Production and shared staging migrations run as part of deployment.

export AWS_PROFILE="<profile-for-the-target-account>"
STACK="dev-<name>"
export DATABASE_URL="$(pulumi stack output database_url_admin -s "$STACK")"

Run migrations:

cd hawk
uv run alembic upgrade head

Deploying import model discovery

Migration 5b61c7e80429 can run online through the normal deployment migration step, before the new importer/API starts. It adds two eval columns and a retained-group branch to the existing group function, with no historical backfill or table rewrite. It needs a brief exclusive lock and has a five-second lock timeout; investigate blocking transactions and retry if needed.

  1. Land the import resume fix and the download metadata fix first and use release code containing all three changes. Apply the migration before starting the new importer/API, then verify the installed revision and both new columns.
  2. Before new importer workers start, pause and drain old workers, including the live consumer. Finish old Batch jobs or cancel and resubmit them against the new job definition; queued jobs can still reference the old image. An old worker resuming a tracked import can write samples without recording their model names, leaving later retries to skip samples whose models were never recorded.
  3. Deploy compatible Middleman model-group responses before the strict Hawk API. Coordinate Control Tower and other ordinary callers' declarations, then deploy the importer/API and resume writers. Verify a restricted test import in both gates.

For rollback, pause writers before reverting importer code and review tracked partial imports before old workers resume them. Retain published policies, both columns, and the group function's retained-group branch; dropping that floor can reopen access. Keep the API reader fallback disabled on rollback: older code can replace stored groups on a read and undo published restrictions.

Backfilling folder model policies

The viewer, API, and S3 Object Lambda decide who can read an eval set or scan folder from its .models.json policy. Policies written before the importers published them list only the models declared at launch, so they can be looser than the warehouse. scripts/ops/backfill_model_policies.py copies the model names and groups the warehouse recorded into those policies.

The backfill is union-only: it adds names and groups and removes none, with one exception: --kind scans drops a stale __unresolved__ marker (see below). So on a folder that already has a policy it can only narrow who can read it, except that a scan folder whose scans have all resolved becomes readable again to holders of its remaining groups, as the importers would have left it. A folder with no policy gets one only when the warehouse has groups for it. That widens access, from no reader to holders of those groups (who can already see the rows in the warehouse), so the report lists these folders separately as would create / created.

Modes, selected with --kind:

  • evals (default): each eval set's eval.observed_model_names and eval.model_groups into evals/<id>/.models.json. Names are written together with the groups they resolve to. Names that match no Middleman model are reported as unresolved and never written.
  • scans: each scan run folder's names (its source evals' observed_model_names) and scan.model_groups into scans/<run>/.models.json. A write run takes the importers' per-folder lock, so it can run while importers are live.
  • scan-usage: adds the groups of the models each scan's scanners actually called (its stored scan_model_usage keys) to scan.observed_model_groups and scan.model_groups, then republishes the scan folders. Only groups are written, since the keys are short model names. Keys that match no Middleman model are only reported, so review them in the dry run. It can't see scanners that return a list of results: their usage was never stored (Scout drops it from each result, and a run that returned none leaves no row).
  • scan-parquet-usage: reads only the scan_model_usage column of every scanner parquet of every scan (found in S3, not through warehouse rows) and records the Middleman names in scan.observed_model_names, as the scan importer does, so their groups restrict the scan, then republishes the scan folders. A name Middleman doesn't know hides the scan until it does, unless the folder's .models.json names it (a dynamic-catalog model). A scan missing a parquet for a scanner its _scan.json names, without a _summary.json entry accounting for it, or with an unreadable one, is reported as failed and left unchanged. A parquet written before Scout recorded usage takes its scanner's usage from _summary.json, or is reported if that has none. A scanner with no parquet that _summary.json says never ran is taken to have called nothing, and is listed in the report (no runs:) so it can be checked.

A scan result imported before the sample it scanned is marked __unresolved__, a group no one holds, and the backfill carries that marker into the folder policy: the scan folder is hidden from everyone until the scanned transcripts are imported, at which point the importers remove it. --kind scans also removes it from a folder where no scan still carries it, which reopens that folder to holders of its other groups. The dry run counts folders that would gain the marker; check that count before a write run.

A scan whose scanners called a model that matches no middleman.model row is marked __unresolved__ too, and its import fails for retry. Models from Middleman's dynamic OpenRouter, Fireworks and Together catalog never get a row, so the scan importer accepts one when the scan folder's .models.json names it. Launch and API import write only names Middleman confirmed, and the importers' publish adds only names that match a row. The importer records such names in scan.confirmed_model_names and requires all of the folder's groups for them.

Run it as a database role that bypasses row-level security: BYPASSRLS, or a role that inherits rls_bypass (such as inspect-importer). Under RLS it would silently skip the restricted rows it exists to protect, so it refuses to start.

After deploying the importers that publish scan folder policies, run the modes in this order: --kind scans, then --kind scan-usage, then --kind scan-parquet-usage, then --kind evals. Run every mode with --dry-run first and review the report before writing:

DATABASE_URL=postgresql://... uv run python scripts/ops/backfill_model_policies.py \
    --bucket <bucket> [--kind scans|scan-usage|scan-parquet-usage] --dry-run
DATABASE_URL=postgresql://... uv run python scripts/ops/backfill_model_policies.py \
    --bucket <bucket> [--kind scans|scan-usage|scan-parquet-usage] [--id <folder-id> ...]

See scripts/ops/README.md for the full flag reference.

Creating a New Migration

After changing the SQLAlchemy models in hawk/core/db/models.py, use an isolated development database—never staging or production—to generate the migration:

cd hawk
uv run alembic revision --autogenerate -m "description of change"

Validate the complete migration chain against the disposable test database:

cd hawk
uv run pytest tests/core/db/test_alembic_migrations.py

Schema Conventions

  • All tables have a pk UUID primary key, and created_at/updated_at timestamps
  • All timestamps are timezone-aware and stored in UTC
  • Model names are singular