Security¶

This page covers Hawk's security architecture, access control, audit logging, and optional AWS security services.

Authentication¶

Hawk uses OIDC (OpenID Connect) for interactive user authentication. JWTs are validated at authenticated HTTP boundaries, including the API server, Middleman (LLM proxy), and token broker. AWS-native integrations use IAM and resource policies instead. The web viewer is a static single-page app: it performs the OIDC login in the browser, and every request it makes to Hawk's data APIs is validated by the API. API-issued presigned S3 URLs act as short-lived bearer capabilities for direct downloads.

Default: Cognito¶

When no OIDC provider is configured, Hawk creates a Cognito user pool automatically. Users are managed via the AWS Console or the helper scripts.

Managing Users¶

scripts/dev/create-cognito-user.sh <stack> user@example.com

Managing Model Access Groups¶

Cognito automatically includes group memberships in the cognito:groups claim of access tokens. Create groups matching the model groups configured in Middleman:

# Create a model access group
scripts/dev/manage-cognito-groups.sh <stack> create model-access-openai

# Add a user to the group
scripts/dev/manage-cognito-groups.sh <stack> add-user model-access-openai user@example.com

# List all groups
scripts/dev/manage-cognito-groups.sh <stack> list

Users who aren't in any group fall back to hawk:defaultPermissions (default: model-access-public), which grants access to models that belong to the group model-access-public.

External OIDC Provider (Okta, Auth0, etc.)¶

For production deployments, we recommend using your organization's identity provider. Use the autodiscovery script to generate Pulumi config from your issuer URL:

uv run python scripts/dev/discover-oidc.py <your-issuer-url> <your-client-id> <your-audience>

This prints the full set of hawk:oidc* config values to add to your Pulumi.<stack>.yaml. See Pulumi.example.yaml for the complete list of OIDC settings.

OIDC App Requirements¶

Your OIDC application must:

Use PKCE (Proof Key for Code Exchange) — no client secret
Support the authorization_code grant type
Include these redirect URIs:
- http://localhost:18922/callback — Hawk CLI login
- https://viewer.<your-domain>/oauth/complete — web viewer login
- http://localhost:5173/oauth/complete — web viewer run locally against a deployed API (see Using a deployed API server)

Required JWT Claims¶

Hawk extracts permissions from the permissions claim (IdP group-derived) and cognito:groups. It also reads scp, but because those are scopes the client requested — which an IdP may grant to anyone — only public model-access values (model-access-public, model-access-public:read) are honoured from it; any other scp value grants no permission. The claim value can be either a JSON array of strings or a space-separated string:

{
  "sub": "user123",
  "iss": "https://login.example.com/oauth2/default",
  "aud": "your-audience",
  "permissions": ["model-access-openai", "model-access-anthropic"]
}

Or equivalently:

{
  "permissions": "model-access-openai model-access-anthropic"
}

The group names must match the groups assigned to models in Middleman (see Model Groups below).

No permissions claim?

If the JWT carries no honoured permission — no permissions or cognito:groups claim and no allowlisted scp value — Hawk falls back to hawk:defaultPermissions (default: model-access-public). A scp claim holding only non-public scopes is filtered to nothing and counts as no permission, so it triggers this fallback too. This is how ungrouped Cognito users get access without custom claims.

Existing deployments upgrading in August 2026

When you upgrade an existing deployment to this release, model-access-public becomes an ordinary group — a user only sees public-model eval/scan data if they hold it. There are two ways a user can hold it, and most deployments are already covered:

From default permissions — hawk:defaultPermissions defaults to model-access-public and is applied to any user whose token carries no permissions/scp/cognito:groups claim. So claimless users (e.g. ungrouped Cognito users) keep access automatically, no action needed.
From their own claim — users whose tokens do carry explicit model-access-* claims must include model-access-public among them. The default is not merged into existing claims, so add it in your IdP for anyone who should see public data.

The only users who lose access are those whose explicit claims omit model-access-public — grant it to them.

Setting Up Your Identity Provider¶

The exact steps vary by provider, but the general approach is:

Create an OIDC application in your IdP (Okta, Auth0, Entra ID, etc.) with PKCE and the redirect URIs above.
Create groups in your IdP for each model access level you need (e.g., model-access-openai, model-access-anthropic). These must match the group names you assign to models in Middleman.
Add a custom claim named permissions to your access tokens that includes the user's group memberships. Most IdPs support this via:
- Okta: Custom claim on an authorization server using a Groups expression
- Auth0: Post-login Action that adds groups to the access token
- Entra ID (Azure AD): App roles or group claims in the token configuration
- Keycloak: Protocol mapper for group membership
Configure Hawk with your IdP's client ID, audience, and issuer URL in your Pulumi stack config.

Access Control¶

Model Groups¶

Each model configured in Middleman belongs to a model group. Permissions separate access to stored results from using the model:

Permission	Read/download results	Import/edit results	Call models
`model-access-example:read`	Yes	No	No
`model-access-example:write`	Yes	Yes	No
`model-access-example:execute`	No	No	Yes
`model-access-example`	Yes	Yes	Yes
No matching grant	No	No	No

Set these grants in the identity provider's trusted permissions or cognito:groups claim. Model registry and result-policy group names remain unsuffixed. Permissions are checked per group; access to one model group does not grant access to another.

Grants combine: :read plus :execute permits running evals and reading results, without editing results. :execute alone permits model calls, but cannot launch Hawk runs or read saved results for that group. Unsupported suffixes grant nothing.

Scans require read access to their source results and read plus execute access to their scanner models. Their results retain both groups' restrictions. Existing ownership requirements are listed in the endpoint audit.

Warehouse evaluation results are restricted by the models declared or observed in the evaluation. Folder and pod reads use their own model-group checks, as described in the endpoint audit.

Result access also controls visibility of usage, including user/team/job/task labels and estimated spend. Hawk admins can normally see usage for all models, but prices remain restricted to their readable model groups, and job and task labels to runs whose named code groups they hold. With hawk:publicModelsOnly, usage is limited to public models even for admins.

For lockdown deployments, Middleman's authenticated /public_models_info endpoint exposes public-model metadata independently of the caller's groups, with secret details hidden. It does not grant permission to run those models.

Models are assigned to groups by Middleman admins when configuring the model. For example, invoking a model with group: "model-access-openai" requires model-access-openai or model-access-openai:execute in the user's JWT permissions claim. The model-access-public group is the default for models intended for all users.

How a model in a log is matched to its group¶

The warehouse records the model's Middleman public name: the importer keeps the part of the log's model URI that Inspect sends to the provider, dropping the Inspect provider and any cloud service segment, but keeps an aggregator prefix, so anthropic/claude-opus-4-6 and anthropic/bedrock/claude-opus-4-6 are stored as claude-opus-4-6 and openrouter/openrouter/z-ai/glm-5.2 as openrouter/z-ai/glm-5.2. Each Inspect provider's rule is listed in INSPECT_PROVIDER_NAMING (hawk/core/providers.py); a provider not listed there (a third-party one) keeps everything after its name, including any service segment it would not send. Rows imported by older releases hold the bare leaf instead (glm-5.2) until the log is re-imported, and a run that did not go through Middleman can yield a name Middleman registered with a prefix. Group lookup therefore matches the two spellings against each other, accepting one name as a path suffix of the other, and a model registered either way gates its eval and scan data.

The Inspect name as written is kept beside the public name in *_raw columns (eval.model_raw, scan.model_raw, model_role.model_raw, and a *_raw copy of each model-usage column). They sit on the same rows, so row-level security covers them the same way. Several Inspect names can map to one public name; the public-name usage columns sum their usage.

A suffix has to start at a / boundary on both sides: claude-3 matches anthropic/claude-3, but secret is a different model from anthropic/claude-secret. The importer follows the same rule when it picks a name, so the name it stores is always a whole-segment suffix of the registered one.

Upgrading from a release before this matching existed

Hawk previously compared the two names for exact equality. A model registered with a prefix never matched what the warehouse stored, so its evals and scans came out with no model group — and a resource with no model group is unrestricted. Those rows were readable by everyone.

The migration recomputes the cached groups, so on upgrade they become restricted to the model's group. Users who need them must hold that group; grant it before upgrading if they don't. Nothing is deleted — granting the group restores visibility.

Downgrading that migration restores the old matching for anything imported afterwards, but deliberately leaves the recomputed groups in place — rolling back must not re-publish the rows the upgrade restricted. To deliberately return those rows to their old, unrestricted state, clear their model_groups yourself after downgrading.

Declared and observed model dependencies¶

Warehouse restrictions include registered primary models, model roles and sample model calls, plus retained groups from import-time discovery. Unknown names add no restriction to the warehouse's model lookup. Empty groups pass warehouse, folder, pod and monitoring checks; a missing folder permission file is denied.

Running evals and pending, importing or failed imports retain their launch-time groups until terminal import succeeds. Scan reads also check current source-eval restrictions. Missing source transcripts carry __unresolved__, which denies ordinary warehouse access and cannot be granted; database administrators retain their RLS bypass.

Endpoint audit¶

Surface	Model permission	Ownership
`/meta/eval-sets`	All sibling evals, including those hidden by filters or row-level security	None
`/meta/jobs`	Launch groups and all imported results	None
`/meta/evals`, `/meta/scans`, sample metadata/scores/events/timeline and warehouse search	Warehouse groups; scans also check current source evals	None
Config, sample content, viewer logs/downloads, scan exports, transcript grep, eval-set status/stream and human-eval metadata	Folder groups; batch downloads check every folder	None
LLM transcript search	Read access to the transcript and execute access to the search model; cross-provider disclosure checks apply	None
Runner logs, traces and monitoring	Job/pod groups and current warehouse restrictions, including declared and imported scan sources	None
Eval stop/delete; scan delete	No separate content check	Recorded owner or admin
Eval/human relaunch, including config-pinned IDs	Read and execute access to every recorded folder group; validate requested models before launch	Owner or admin
Scan resume	Read access to existing results and sources; read and execute access to current scanner groups	Owner or admin; active or unverified releases are refused
Fresh eval, human-eval and scan launches	Read and execute access to requested models; scans also require source read access	Reserve owner before launch writes; retain after failure
Eval/scan imports	Write access to all resulting model groups and existing content on updates	Reserve owner at allocation; owner/admin for uploads and finalization
Sample score overwrites and invalidation changes	Write access to folder groups	Run/import owner or admin
Append info/score events	Write access to folder groups	None
Relay, ACP, attach and browser approvals (`/approvals/{job_id}`, HTTP preflight and WebSocket)	Full access (read, write and execute) to every relevant folder/pod group and a write-capable token	None
Code-access rule preview and source registry (`/code_rules/preview`, `/code_rules/repos`, `/code_rules/repos/{pk}/code`)	None; no code-group filter, so admins see sources and code items outside their code groups	Admin only
Your code groups' rules (`/meta/code-groups`)	Lists only the prefixes of rules that require a code group the caller holds, never sources: a source's name comes from runs whose models the caller may not read	None
Runner Git tokens (token broker `/github`)	The job's own pod-bound job token, not a user token; tokens only for repositories no rule restricts or the run declared at launch, where the launcher's code access was checked	None

Reading does not require ownership. Warehouse checks do not update folder permission files or pod groups, so access can differ between these surfaces. Viewer file writes/deletes are disabled; unsupported scan-viewer data paths return 400.

Pinned IDs: IDs with warehouse results or human assignments but empty S3 folders are refused, including for admins.
Legacy ownership: Native runs without a launch record require an admin with content access for resume or destructive edits; imports without an owner record require the same for updates or destructive edits. An admin resuming a legacy native run becomes its recorded owner; stop/delete still require a launch record.
Imports: Owner records live in the API-only import-owners/ prefix. Admins cannot finalize another uploader's stamped files without reuploading with their own attribution.
Literal search progress: searched_through reports only the creation time of a row the caller can read. A page that examined only unreadable rows leaves it unchanged while the cursor advances, which shows that the span between two readable rows holds only unreadable rows. Hawk accepts this: it discloses no row identity, content or timestamp, and empty pages that still return a cursor already show that unread rows exist.
Failed scan creation: If config or permission files are missing, resubmit with hawk scan run to get a new run ID. The retained owner record alone does not make a scan resumable.

API-only helpers use the reserved hawk_api database role, granted to the API login (the first warehouseReadWriteUsers entry). Warehouse readers retain the scan helpers needed for row-level security.

Models discovered during import¶

New imports retain original model identities from the primary model, roles, eval and sample usage, and model events as work commits. On completion, the importer adds their resolved groups to warehouse restrictions and the existing .models.json. Both retain earlier restrictions; replacing content never automatically declassifies it. Native and imported scans inherit their settled sources' groups plus scanner groups.

Discovery adds no sample reconstruction pass on retry: previously committed names are read from the eval row. Existing header normalization reads remain unchanged. Rows with observed_model_names IS NULL are legacy imports and receive no automatic repair; an accepted forced full reimport opts them in. No historical backfill runs.

This is an eventual backstop. Undeclared calls may expose data before completion, so declare models before launch. Unknown models or missing/malformed policies fail publication through the existing importer retry path; they require operator attention if retries exhaust. Concurrent scans and source writes receive no additional guarantee. See the deployment procedure.

Provider-discovered models¶

When Middleman loads its database-backed registry, it also records models found in provider feeds and their model-access groups. Stored names and their original groups are retained even if a provider stops listing them. The warehouse resolves these names with the same suffix matching as administrator-registered models.

Discovery adds missing restrictions to matching existing evals and scans, including scans that read those evals. It preserves every cached group and the __unresolved__ marker that hides unfinished scan imports. Registry reloads retry stored names, so an import that overlaps the first discovery is repaired on the next successful load. A database failure is logged and does not stop Middleman from serving its registry; repair waits for a successful retry.

This backfill targets full names and their path suffixes in eval, model-role, sample-model, and scan records. It does not search sample usage JSON or the reverse suffix direction; those cases wait for their normal import refresh. Discovery does not rebuild jobs or .models.json, and changing an existing model's group still requires deliberately rebuilding the affected caches.

Read-only scopes¶

Credentials whose model grants are all :read cannot make mutating requests (403). Use these for machine or agent tokens that only need to read results.

How Group Membership Flows¶

flowchart LR
    IdP["Identity Provider<br/>(Okta, Auth0, Cognito)"]
    JWT["JWT with<br/>permissions claim"]
    API["Hawk API"]
    MM["Middleman"]
    TB["Token Broker<br/>(Lambda)"]
    S3["S3 (eval logs)"]
    DB["PostgreSQL<br/>(RLS)"]

    IdP -->|"issues"| JWT
    JWT -->|"validates"| API
    JWT -->|"validates"| MM
    JWT -->|"exchanges for<br/>scoped credentials"| TB
    TB -->|"scoped access"| S3
    API -->|"RLS enforces<br/>group membership"| DB

Identity Provider issues JWTs with model group memberships in the permissions claim
Middleman validates the JWT and checks the user's groups before routing model API calls
Token Broker validates the user's model group permissions, then exchanges the JWT for scoped AWS credentials tied to a specific job via AWS session tags
PostgreSQL RLS (Row-Level Security) restricts database queries to evaluation results the user is authorized to see

One caller does not come from the identity provider: a stack can let named GitHub Actions workflows call Middleman with GitHub's OIDC token, and their model groups then come from the stack's configuration rather than an IdP claim (GitHub Actions workload identity).

Token Broker Job Identity¶

The broker always validates the user's access token. Runners also send a projected Kubernetes ServiceAccount identity token so the broker can bind the request to a specific job:

Factor	Header	Proves	Issued by
User access token	`Authorization: Bearer`	Permission — the caller's model grants authorize the requested operation	Your OIDC provider
Job identity token	`X-Hawk-Job-Token`	Identity — the caller really is the job whose `job_id` it requests	The EKS cluster's OIDC issuer

The job identity token is a Kubernetes projected ServiceAccount token mounted into the runner pod, scoped to the hawk-token-broker audience and rotated by kubelet. The broker validates its signature against the cluster's JWKS endpoint, then checks that the token's sub equals the full system:serviceaccount:<runner-namespace>:<runner-service-account> it derives for the requested job, and that the token carries a pod binding.

With hawk:requireJobToken: "true", both tokens are required and the user token alone is insufficient. This prevents a runner with broad model permissions from requesting credentials session-tagged for another eval set's job_id and gaining access to that eval set's S3 prefix. The default is currently permissive, as described below, so deployments do not get this two-factor guarantee until they explicitly enable enforcement.

Output credentials without verified worker identity require write and execute access to every result group. Workers without those grants need a matching, pod-bound job token even when requireJobToken is false.

Enforcing (`requireJobToken`)¶

hawk:requireJobToken is false (permissive) by default:

Permissive — a missing, invalid, or mismatched job token is logged as an audit warning and emits a JobIdentitySkipped CloudWatch metric, dimensioned by error_type (missing, invalid, mismatch, unbound, error). Credentials may still be issued if the caller independently satisfies the required model permissions above.
Enforce (true) — the same conditions return 403 and emit JobIdentityDenied with the same dimensions.

Enforcing kills in-flight jobs launched before the chart change

Only runners created after the Helm chart change have the projected token volume. Flipping requireJobToken to true makes every older runner fail at its next credential refresh — which, because credentials last an hour, can be up to an hour after the flip and appears as a mid-eval S3 failure.

Rollout:

Deploy with requireJobToken: false and let the existing jobs drain.
Watch the "Token broker: job identity (permissive skips)" widget on the Hawk platform overview dashboard. missing decays toward zero as pre-rollout jobs finish; it should stay at zero for jobs launched after the deploy. A persistent non-zero error usually means the broker's JOB_TOKEN_* configuration is wrong rather than that a runner misbehaved.
Once no newly launched job reports missing, and no job is still running from before the chart change, set requireJobToken: true.

mismatch is the one reason that is never benign — it means a caller asked for a job it does not hold the identity token for. In permissive mode credentials can still be issued to callers with sufficient direct permissions. When hawk:enableProdAlarms is true, the matching <env>-hawk-token-broker-identity-mismatch-permissive or <env>-hawk-token-broker-identity-mismatch-denied alarm fires on the first occurrence. Subscribe a receiver to their shared SNS topic before starting the rollout. The flag controls alarm creation regardless of the stack name; alarms are not created merely because a stack is named prd.

In enforce mode the broker refuses to start a request with empty JOB_TOKEN_* configuration, and pulumi up fails when requireJobToken is enabled without a resolvable EKS OIDC provider URL — an empty issuer would otherwise 403 every runner while looking correctly configured.

Runner Git access¶

With a GitHub App configured (see new environment), runners never hold a long-lived Git credential. Each runner's GIT_CONFIG_* routes GitHub's challenge for a private repository to python3 -m hawk.runner.git_credential, which asks the token broker's /github with the pod's projected ServiceAccount token, the one it already sends for AWS credentials. For this the broker accepts only that job's own pod-bound token (issuer, the job's full sub, pod binding), whatever requireJobToken says. It then mints an installation token for that one repository with contents: read, valid for an hour, and Git downloads directly from GitHub. Public repositories never reach the broker.

The broker mints only for repositories the App is installed on, and only if no code-access rule restricts the repository or the run recorded it at launch, even with an incomplete inventory. The database decides, through code_policy_git_source_access, which the broker's narrow login may call. A recorded repository grants nothing else in its group: a run's code groups restrict who reads it and can grow after launch. Launches check that the user may use each restricted repository the run declares; the broker then keeps out restricted repositories it didn't declare, such as a task's private dependency. Each token is logged with the job, the repository and why it was allowed, and no two requests share one, since any holder can revoke it. Job tokens are checked offline, so a copied one works until it expires, up to an hour after it was issued, as it does for the AWS credentials; each GitHub token works for an hour.

For repositories no rule restricts, the App's installation scope, not Hawk, bounds what runs can read: every user who can launch a run can read every such private repository the App is installed on. The App's private key can mint tokens for all of them, with every permission the App holds, so register it with Contents: read-only alone. It lives only in <env>/hawk/github-app-key, whose policy lets only the token broker read it, plus any break-glass principals listed in hawk:githubAppKeyBreakGlassPrincipals. The broker re-reads it every five minutes, so to rotate it, generate a second key in the App's settings, write it to the secret, wait five minutes, then delete the old key.

Without the App (hawk:githubAppId unset), runners keep receiving the <env>/inspect/api-git-config credential as before. Every runner could copy that credential, so once no stack hands it to runners, write back the secret's placeholder and revoke the token on GitHub. Keep every key, with GIT_CONFIG_COUNT at 0 and the rest blank: the API task references each key, and ECS refuses to start it if one is missing.

Administrative Roles¶

Hawk has one administrative role: Middleman Admin. Admins can:

Create, update, and delete model configurations
View all models regardless of group membership
Manage provider API keys

To grant admin access, add a boolean claim to your OIDC access tokens:

Claim	Purpose
`https://middleman.metr.org/claims/admin`	Full admin access (production + non-production)
`https://middleman.metr.org/claims/dev-admin`	Admin access for non-production environments only

The dev-admin claim is only accepted when the Middleman environment variable MIDDLEMAN_ACCEPT_DEV_ADMIN=true is set (which Hawk configures automatically for non-production stacks).

Example JWT with admin access:

{
  "sub": "admin-user",
  "permissions": ["model-access-openai"],
  "https://middleman.metr.org/claims/admin": true
}

In your IdP, create a group (e.g., middleman-admins) and configure a custom claim that emits true when the user is a member of that group.

Hawk Admin¶

Hawk admins can stop, delete and resume other users' runs, update imports and sample scores/invalidation, manage human-registry entries, and author code-access rules. Admin status bypasses ownership checks, not model access or named code groups; on reads it passes missing or incomplete code inventories (a run with no policy and no rows is refused). To author rules, admins can read the whole code registry, sources outside their code groups included: every inventoried source and its code items' names, subdirectories and refs. The endpoint audit lists the checks and legacy exceptions.

There are two OR-ed sources of admin status; both are disabled by default:

Config key	How it works
`hawk:hawkAdminClaim`	Name of a JWT claim. When the claim is present and its value is boolean `true`, the bearer is treated as a Hawk admin.
`hawk:hawkAdminPermissions`	List of permission/group names. If any entry matches a value in the token's `permissions` or `cognito:groups` claim, the bearer is treated as a Hawk admin. Values from `scp` cannot confer admin — only public model-access scopes are honoured from `scp`, and those may not be admin names.

hawkAdminPermissions entries must be disjoint from defaultPermissions: tokens that carry no permission claims have the defaults substituted in, so any overlap would make every such caller an admin. The API refuses to start if it detects an overlap. The entries must also not be OAuth scopes that users can self-request from your IdP.

Ownership overrides emit a warning-level admin_override event with the action, actor and recorded owner (null when unavailable). Job events include job_id; human-registry events include human. Actions are stop, delete, resume, edit, update_import or update.

Browser approval WebSockets validate the API session cookie/JWT and viewer Origin before upgrading, and close when the credential expires. Content access and write capability are rechecked before sending each decision. approval_decision logs record the actor, run, session, tool call, offered choice and delivery or application status; tool arguments and credentials are not included. Browser approval sessions do not expose general ACP commands or arbitrary relay ports. These audit records identify an account, not proof of human intervention.

Browser approval review requires an HTTPS/WSS relay outside localhost because the API forwards the reviewer's bearer token to the relay. HTTP-only testing deployments using hawk:skipTlsCerts: "true" must enable TLS to use this feature.

The adapter parses runner-controlled ACP messages, capped at 2 MiB each. A review retains at most 10,000 samples and 64 pending approvals, with a 4 MiB total limit on serialized pending approval data. Exceeding a limit closes the review without answering pending requests; use hawk acp for runs exceeding these limits.

Runner pods receive the launching user's access token. An admin's token lets task code exercise these same powers—including resume, import updates and destructive sample edits—while model-access checks still apply. Prefer separate day-to-day and admin identities, or short-lived admin membership.

Example: using hawk:hawkAdminClaim with an Okta custom claim:

{
  "sub": "admin-user",
  "permissions": ["model-access-openai"],
  "hawk-admin": true
}

pulumi config set hawk:hawkAdminClaim hawk-admin

For Cognito deployments, hawk:hawkAdminPermissions is simpler — create a Cognito group (e.g. hawk-admin) and add it to the config:

pulumi config set --path 'hawk:hawkAdminPermissions[0]' hawk-admin

Code-access rule authoring¶

Administrators can create and edit code-access policy through /code_rules/. Users need every required code group and model group to submit a run or read its results. The API and warehouse RLS check code policy in the database; model-folder checks still use .models.json. On reads, Hawk admins pass missing or incomplete inventories but need named code groups; a run with no policy and no rows is refused. Model checks and submission validation still apply. Other readers need dedicated code-access-* permissions; team claims attribute usage. See Code-access rules.

Native-run owners may use the Hawk API to read through their own incomplete code inventory, using Hawk's recorded ownership rather than uploaded log metadata. Named groups, invalid-source restrictions, model permissions and scan-source restrictions remain enforced; missing policies and pending inventory writes remain blocked. This does not grant mutation access or an owner exception for direct SQL reads. Runs without recorded ownership require admin help.

The relay also checks current policy through the API before accepting hawk acp or hawk attach, in addition to requiring model-write permissions.

Inventory covers direct declared dependencies. Transitive dependencies and code fetched during execution remain outside this boundary. Historical runs retain their explicit baseline until backfilled; outside the read exceptions above, missing or incomplete policies deny access. The S3 log reader asks the database's code policy on every read, with no admin or owner exception, and answers 503 when the database or IAM Identity Center cannot answer. The token broker asks the same policy before issuing S3 credentials to anyone but the run's own verified job, and for every scan source.

Sensitive Model Protection¶

Hawk protects sensitive model information through codenames:

Models are given a public_name (codename) that is used everywhere in the system
The real model identifier (danger_name) is only known to Middleman
Evaluation results, logs, and the web viewer only show the public_name
Observability telemetry uses layered protections: model fields and tags use public_name, known credential fields are removed, and recognized model-bearing URL forms are scrubbed. URL and arbitrary-text scrubbing are best-effort, so observability access must remain restricted.

Users without the appropriate model group cannot access evaluation results that used that model.

For models with are_details_secret, the usage price endpoint also hides prices and provider/lab details.

Sandbox Isolation¶

Evaluations run in isolated Kubernetes pods with:

Separate namespaces — each evaluation gets its own namespace for the runner and sandbox pods
Network policies — Cilium network policies block egress to VPC infrastructure (primary subnet CIDR, EC2 IMDS, EKS Pod Identity) from sandbox pods, and can restrict no-internet pods to DNS-only egress
Resource limits — CPU and memory constraints per pod
StatefulSets — sandbox pods auto-restart on failure

Sandbox egress is deny-by-default; internet access is opt-in per task. For how to configure it per evaluation (and the main use case, cyber-related evals), see Sandbox Network Isolation.

Alternative Sandbox Providers¶

While Kubernetes is the default sandbox environment, Hawk's architecture does not strictly require it. EC2-based sandboxing and other providers (e.g., Modal) can be used as alternatives. The sandbox provider is configured per evaluation.

gVisor RuntimeClass and CNI mode¶

Set hawk:enableGvisor: "true" to install the gvisor RuntimeClass. Pods opt into gVisor with runtimeClassName: gvisor; the RuntimeClass uses the pinned runsc runtime. Enabling it changes the default Karpenter node class from Bottlerocket to AL2023 so the runtime can be installed during node bootstrap. The AL2023 node configuration preserves single-process OOM killing for workload containers.

The default hawk:ciliumExclusive: "false" keeps Cilium chained to AWS VPC CNI and retains the default RFC 1918 Cilium pod pool. Exclusive mode makes Cilium the node CNI, uses an RFC 1918 overlay pod pool, and retires the aws-node DaemonSet after Cilium is ready. Treat that as a CNI migration for existing clusters: plan node recycling and a rollback path before enabling it.

Audit Logging¶

Application-Level Logging¶

Hawk API — all API requests are logged to CloudWatch with user identity, action, and resource context (the eval set or sample a request resolved). Query strings are not logged, with one exception: transcript search (/meta/search) records the first 256 characters of the search term, so an access review can see what was searched, not only that a search ran.
Middleman — at the default summary traffic-log level, model API calls are logged with user identity, public model name, token usage, and body sizes, but request/response bodies are not retained. At hawk:middlemanTrafficLogLevel: full, bounded raw bodies are archived in S3: known headers and the exact top-level api_key field in a parseable JSON request object are redacted, but other request values and response bodies are not value-scrubbed. Treat this archive as sensitive.
Token Broker — credential exchanges are logged with user identity and requested scope; GitHub tokens with the job, the repository and why it was allowed, since no user token backs them

AWS CloudTrail¶

Hawk does not provision a CloudTrail trail. Configure organization- or account-level trails, destinations, retention, and optional CloudTrail Insights outside this repository. In METR's deployment, AWS Control Tower supplies the organization trail and its existing destinations; METR's private infra-shared repository adopts that trail to configure Insights and selected data events and adds query and monitoring integration. Infra-shared does not create the Control Tower log bucket/log group or control their retention.

VPC Flow Logs¶

Hawk does not provision VPC Flow Logs. If you need network-level audit logging, enable VPC Flow Logs separately and choose a CloudWatch Logs or S3 destination and retention policy that fits your deployment. hawk:cloudwatchLogsRetentionDays (default: 14 days) does not control a separately managed flow-log group. METR's infra-shared deployment enables all-traffic flow logs for staging and production at /aws/vpc/flowlogs/<env> with 365-day CloudWatch retention; production also uses an S3 destination.

Endpoint Protection (CrowdStrike Falcon)¶

Hawk optionally deploys the CrowdStrike Falcon sensor to protect infrastructure hosts. Enable it with hawk:enableCrowdstrike: "true" and a Secrets Manager secret containing your CrowdStrike API credentials (see Configuration).

What's Protected¶

Target	OS / Arch	Installation Method
All EKS nodes (Karpenter)	Bottlerocket and AL2023	DaemonSet via falcon-sensor Helm chart (requires Falcon Images Download scope)
Tailscale subnet router	AL2023 / ARM64	Sensor RPM installed via cloud-init at boot

EKS nodes get the sensor from the DaemonSet and only from the DaemonSet. GPU nodes once also installed it as a host RPM from EC2NodeClass userData; they no longer do, because a node cannot run both — the containerized sensor exits immediately on a host that already owns /opt/CrowdStrike, leaving its pod in CrashLoopBackOff. Hosts outside EKS have no DaemonSet to rely on and still install the RPM.

The DaemonSet pulls a container image from registry.crowdstrike.com, which requires the Falcon Images Download: Read scope — part of the Falcon Cloud Security with Containers add-on. The subnet router downloads its RPM from the CrowdStrike API at boot using the Sensor Download: Read scope.

Tailscale ZTA Integration¶

With the Falcon sensor on the subnet router, you can enable CrowdStrike ZTA with Tailscale to gate tailnet access based on device posture. This requires a separate API client with Hosts: Read and Zero Trust: Read scopes, configured in the Tailscale admin console under Device Posture Integrations. Note that ZTA scores apply to the router instance itself, not to traffic routed through it.

Runtime feature flags¶

Operators can change runtime flags without a merge, deployment, or API restart. Each deployment gets an AWS AppConfig feature-flags profile from Pulumi: application <env>-hawk-api-feature-flags, environment <env>, configuration profile feature-flags. The API reads the deployed configuration on demand and caches it for ten seconds, so a toggle reaches every replica within about that long. Requests that already read the flag finish under the previous value, and changing a flag does not stop running jobs.

How the cross-lab rule decides¶

Hawk stops a model from reading non-public transcripts produced by models whose read policy doesn't allow it. By default that is any model another company serves. The check runs for standalone scans (create and resume), online scans in eval sets (create and resume), and LLM transcript search. The receiving model is the scanner or search model; the source models are the models whose transcripts it would read.

Each non-public source model's read policy, recorded in Middleman (see Recording who may read a model's transcripts), decides which receiving models may read it. Middleman returns the policies with the groups and providers from /model_groups; a model with no policy has the default.

Mode	Receiving models allowed
`provider` (default)	Those served by the source model's provider
`any`	All
`tags`	Those whose `reader_tags` include one of the policy's `tags`
Any other	None: a mode this Hawk doesn't know is refused

A policy with zdr_only additionally requires a receiving model whose is_zdr is set, whatever the mode. Tags and zdr_only can therefore tighten access as well as loosen it: a same-provider model without a matching tag, or without ZDR, is refused. A receiving model's own policy doesn't matter, only its reader_tags and is_zdr. A request is refused if any receiving model may not read any source model.

In provider mode, both sides are judged by the serving provider that Middleman records for the model in its /model_groups response. The provider is the company that receives the data. A model served through OpenRouter or another aggregator has that aggregator as its provider, since sending it data hands that data to the aggregator. Middleman's per-integration lab is not used, so two models served by the same company through different integrations may read each other's transcripts. Providers must match exactly, with no normalization.

A read that a non-default policy allows (any, tags, or a zdr_only the receiving model meets) writes an info-level audit line once the request has passed every check: for scans once the job has launched, for search once the transcript has been found and just before the search model reads it. The line has the caller's email and user ID (the token's subject), the job ID for scans or the sample UUID for search, the receiving and source models, the mode and the matched tags. Like the override line, which follows the same timing, it never names a provider. A scan's lines, and the override line, carry a launch field: launched once the runner's Helm release is installed, or uncertain when the launch failed or the request was cancelled after the launch had begun. Past that point even a failed Helm install may leave a runner reading the transcripts, so those lines are still written, saying the read may not have happened. A launch refused before it begins writes none. Error messages name only the refused source models, never a provider, mode or tag.

Each model's read access from Middleman holds its policy as a source (policy: mode, tags, zdr_only) next to what it holds as a reader (reader_tags, is_zdr). Every field inside policy restricts who may read the model's transcripts, so a source whose policy sets a field this Hawk doesn't know (to a non-empty or true value) is refused like an unknown mode: a newer Middleman's field may restrict reads in a way this Hawk can't check. Unknown fields outside policy describe the model as a reader and are ignored. A model's read access entry that this Hawk can't parse (a known field of the wrong type, for example) is refused as a source and holds no tags and no ZDR as a reader; the rest of the response still applies, and Hawk logs a warning naming the model.

Hawk caches /model_groups results for 15 minutes, so a policy change in Middleman can take up to 15 minutes to apply.

Models are looked up by their Middleman public names. For scans, the inspect provider prefix in a config (for example openai-api/openai-chat/...) is not used, because it only picks the wire format, while Middleman decides where the request goes. Transcript search looks up the search model and the sample's models by their Middleman public names, taken from the sample's recorded model calls. For older data without them, it falls back to the names stored in the warehouse.

A standalone scan's source models are normally every model of its source eval sets: the names in each eval set's .models.json and the models the warehouse recorded for its evals. A scan in which every scanner's filter is only transcript_id conditions is judged by the models recorded for the evals that hold those transcripts instead, so a scanner can read one provider's transcripts from an eval set that also ran another provider's models. The check works per eval, not per sample: every model an eval declared or any of its samples called counts for each of its transcripts. The narrower check applies only when the warehouse holds every requested transcript in a source eval set, each of those evals' imports succeeded with model names recorded, and no transcript is still being ingested live. Otherwise the whole source eval sets are judged, as they also are when resuming a scan that already has results: it continues from its own snapshot of the transcripts, not the warehouse's files for them. Access permissions don't change: a scan still requires the groups of every model of its source eval sets.

Public source models are exempt whatever their policy and need no provider, and a scan with no receiving model (one that uses only non-LLM scanners) is not checked. A model with no recorded provider gets its lab's default from Middleman, so only models on labs without one (the dummy test labs) have none. Providers are needed only where they are compared: if a non-public source model in provider mode has no provider, or one is present and a receiving model has none, the request is refused with a 422 that names those models. Source models in other modes need none. The cross-lab override does not apply to this error; an administrator has to record the missing provider in Middleman. A deployment therefore needs a Middleman that returns providers from /model_groups.

LLM transcript search adds one rule: a transcript whose non-public provider-mode models span more than one provider can't be searched by any model, since no model shares all their providers. Its models in other modes are judged by their own policies, so a transcript of such models from several providers can be searched by a model they all allow.

Cross-lab scan overrides¶

allow_sensitive_cross_lab_scan_override controls whether the API honors the existing allow_sensitive_cross_lab_scan request field (hawk scan run/resume --allow-sensitive-cross-lab-scan). It applies to standalone scan creation, scan resume, and eval creation/resume with an online scan.

Runtime flag	Request asks for override	Behavior for non-public cross-lab transcripts
Enabled (default)	No	Blocked with 403
Enabled (default)	Yes	Allowed, subject to normal model and transcript permissions
Disabled	Either	Blocked with 403

If the request asks for an override and the flag store cannot be read, the request fails with a 503 instead; see below.

Scans the read policies allow and public source models keep their existing behavior. This flag does not affect LLM transcript search or change how the read policies decide. It covers every read-policy refusal, whatever its reason, but nothing else: a request refused because a provider is missing stays refused with a 422 whatever the flag says.

An honoured override writes an info-level audit line with the caller's email and user ID, the job ID, and each overridden read: the source model, the receiving model and the reason it would have been refused (provider, tags, zdr, or policy for an unknown or unparseable policy), in the message and in the structured overridden field. Like the policy-grant lines, it is written once the job has launched, with the same launch field. It never names a provider, and the reasons never reach the caller.

Toggling a flag¶

In the AWS console: AWS AppConfig → Applications → <env>-hawk-api-feature-flags → Configuration profiles → feature-flags, flip the flag, Save new version, then Start deployment into the <env> environment with the AppConfig.AllAtOnce strategy. The same two steps from the CLI, using the IDs shown in the console:

cat > flags.json <<'JSON'
{"version": "1",
 "flags": {"allow_sensitive_cross_lab_scan_override": {"name": "allow_sensitive_cross_lab_scan_override"}},
 "values": {"allow_sensitive_cross_lab_scan_override": {"enabled": true}}}
JSON
configuration_version=$(aws appconfig create-hosted-configuration-version --application-id <app-id> \
  --configuration-profile-id <profile-id> --content-type application/json \
  --content fileb://flags.json --query VersionNumber --output text /dev/null)
aws appconfig start-deployment --application-id <app-id> --environment-id <env-id> \
  --configuration-profile-id <profile-id> --deployment-strategy-id AppConfig.AllAtOnce \
  --configuration-version "$configuration_version"

Deployments use the built-in AppConfig.AllAtOnce strategy: the new value is served immediately, then the environment bakes for ten minutes, during which StartDeployment is rejected with ConflictException. To reverse a toggle inside that window, stop the in-flight deployment instead. That rolls the environment back to the previous version, which means every flag in the profile, not just the one you changed:

aws appconfig stop-deployment --application-id <app-id> --environment-id <env-id> \
  --deployment-number "$(aws appconfig list-deployments --application-id <app-id> \
    --environment-id <env-id> --query 'Items[0].DeploymentNumber' --output text)"

Right after Pulumi seeds the profile (the stack's first deploy) there is no earlier version to roll back to, so the first toggle may have to wait out that bake. The bake is kept deliberately: it is what lets AppConfig roll back automatically if CloudWatch alarms are ever attached to the environment.

Every flag declares a default in the hawk_flags package: the value in effect until an operator changes it. A flag that is absent from the deployed configuration, or whose enabled value is not a JSON boolean, uses that default, as does a deployment with no HAWK_API_FEATURE_FLAGS_* settings. If AppConfig cannot be read (access denied, throttling, nothing deployed yet), a request that asked for an override fails with a 503 and should be retried. Requests without an override, and requests with no cross-lab violation, never read the flag.

Pulumi seeds the profile with every flag at its default when its AppConfig resources are first created. Later updates ignore changes to the seed content and the seed deployment's version, strategy, description and encryption key, so those edits do not replay the defaults over operator toggles. A flag added to Hawk later is absent from an existing environment's profile and runs at its default there; to toggle it, add it to the profile in the console first. Changing a flag's default in code likewise reaches only new profiles and environments where the flag is absent: wherever an operator's version carries the flag, its deployed value stands.

This protection does not cover forced replacement or recreation after losing Pulumi state. Replacing the AppConfig application, environment or profile also initializes the new target with the defaults. Preserve the current flag values before these operations and restore them afterwards if needed.

Upgrade behavior: the flag defaults to enabled, so upgrading changes nothing for callers. To turn the override off, deploy a disabled version once the AppConfig profile exists (after the first deployment of this version). Until every API replica has upgraded, older replicas still honor per-request overrides independently of this flag, so a disable takes full effect only once the rollout completes.

AWS Security Services¶

In METR's deployment, AWS security services (GuardDuty, Security Hub, AWS Config, and CloudTrail Insights) are managed by METR's private infra-shared repository, not by Hawk. Other Hawk operators must configure equivalent account- or organization-level services separately.

Recommended Security Configuration¶

For production deployments, consider:

Setting hawk:eksPublicEndpoint: "false" and using Tailscale for private cluster access. If the public EKS API endpoint must remain enabled, set hawk:eksPublicAccessCidrs to restrict it to approved source CIDRs; omitting it preserves AWS's 0.0.0.0/0 default.
Setting hawk:albInternal: "true" to make the ALB private (requires VPN)
Setting hawk:protectResources: "true" to prevent accidental deletion of stateful resources (S3 buckets, secrets, the Datadog log-archive bucket, and the Aurora cluster)

Monitoring & Observability¶

CloudWatch¶

Hawk's application and service logs go to CloudWatch by default. The Hawk-owned groups configured with hawk:cloudwatchLogsRetentionDays retain 14 days by default.

Key Hawk-owned log groups include:

Hawk API logs
Middleman logs
Lambda function logs

In METR's deployment, infra-shared separately owns the GuardDuty and Security Hub finding groups at /aws/events/guardduty/<env> and /aws/events/securityhub/<env>. They retain 365 days and are not controlled by hawk:cloudwatchLogsRetentionDays.

Datadog (Optional)¶

For richer monitoring, Hawk supports Datadog integration:

hawk:enableDatadog: "true"

This enables:

APM — distributed tracing across API, Middleman, and Lambda functions
Log forwarding — CloudWatch logs forwarded to Datadog
Custom metrics — token usage, import counts, evaluation durations
Sensitive data filtering — known sensitive fields and auth headers are removed, while model fields and tags use public names. Model-bearing URL scrubbing is best-effort for recognized provider forms; arbitrary text and unrecognized URLs are not guaranteed to be value-scrubbed. The optional full Middleman traffic archive follows the narrower body-redaction rules described above.

Budget Alerts¶

Monitor AWS spending with budget alerts:

hawk:budgetLimit: "10000"
hawk:budgetNotificationEmails:
  - "team@example.com"
hawk:budgetNotificationThresholds:
  - 80
  - 100

External Dependencies¶

Hawk interacts with the following external services:

Service	Purpose	Required?
Docker Hub	Pull base container images	Yes (login recommended to avoid rate limits)
LLM Providers	Model API calls via Middleman	At least one provider API key required
OIDC Provider	User authentication	Optional (Cognito used by default)
Datadog	Monitoring and observability	Optional
Slack	Budget alerts	Optional
CrowdStrike Falcon	Endpoint protection for EKS nodes and subnet router	Optional
Tailscale	Private VPC, EKS API, and internal ALB access	Optional
Cloudflare	DNS delegation	Optional
GitHub	CI/CD via Pulumi Deploy; with a GitHub App, the token broker mints runners' repository tokens through `api.github.com`	Optional

Network Security¶

TLS by default — external service traffic uses ACM certificates unless an operator explicitly enables the testing-only hawk:skipTlsCerts HTTP mode
Private subnets — EKS nodes, RDS, and ECS tasks run in private subnets with no direct internet access
NAT gateway — the default VPC routes outbound internet access from all private subnets through one shared NAT gateway
Security groups — restrict traffic between components (ALB → ECS, ECS → RDS, etc.)
VPC endpoints — S3 traffic stays within the VPC via a Gateway endpoint
Cilium network policies — pod-level network isolation within Kubernetes

Code source inventory¶

Native launches identify and authorize declared package sources before writing launch files, then commit the accumulated inventory before Helm. Parsing gaps and database failures prevent launch. Ordinary resumes retain earlier sources and model restrictions. An interrupted first launch may require a new ID; cleanup keeps model-access files whenever protected files may survive. Inventory identifies declarations, not every source the runner can install or download.