Security¶

This page covers Hawk's security architecture, access control, audit logging, and optional AWS security services.

Authentication¶

Hawk uses OIDC (OpenID Connect) for interactive user authentication. JWTs are validated at authenticated HTTP boundaries, including the API server, Middleman (LLM proxy), and token broker. AWS-native integrations use IAM and resource policies instead. The web viewer is a static single-page app: it performs the OIDC login in the browser, and every request it makes to Hawk's data APIs is validated by the API. API-issued presigned S3 URLs act as short-lived bearer capabilities for direct downloads.

Default: Cognito¶

When no OIDC provider is configured, Hawk creates a Cognito user pool automatically. Users are managed via the AWS Console or the helper scripts.

Managing Users¶

scripts/dev/create-cognito-user.sh <stack> user@example.com

Managing Model Access Groups¶

Cognito automatically includes group memberships in the cognito:groups claim of access tokens. Create groups matching the model groups configured in Middleman:

# Create a model access group
scripts/dev/manage-cognito-groups.sh <stack> create model-access-openai

# Add a user to the group
scripts/dev/manage-cognito-groups.sh <stack> add-user model-access-openai user@example.com

# List all groups
scripts/dev/manage-cognito-groups.sh <stack> list

Users who aren't in any group fall back to hawk:defaultPermissions (default: model-access-public), which grants access to models that belong to the group model-access-public.

External OIDC Provider (Okta, Auth0, etc.)¶

For production deployments, we recommend using your organization's identity provider. Use the autodiscovery script to generate Pulumi config from your issuer URL:

uv run python scripts/dev/discover-oidc.py <your-issuer-url> <your-client-id> <your-audience>

This prints the full set of hawk:oidc* config values to add to your Pulumi.<stack>.yaml. See Pulumi.example.yaml for the complete list of OIDC settings.

OIDC App Requirements¶

Your OIDC application must:

Use PKCE (Proof Key for Code Exchange) — no client secret
Support the authorization_code grant type
Include these redirect URIs:
- http://localhost:18922/callback — Hawk CLI login
- https://viewer.<your-domain>/oauth/complete — web viewer login
- http://localhost:5173/oauth/complete — web viewer run locally against a deployed API (see Using a deployed API server)

Required JWT Claims¶

Hawk extracts permissions from the permissions claim (IdP group-derived) and cognito:groups. It also reads scp, but because those are scopes the client requested — which an IdP may grant to anyone — only public model-access values (model-access-public, model-access-public:read) are honoured from it; any other scp value grants no permission. The claim value can be either a JSON array of strings or a space-separated string:

{
  "sub": "user123",
  "iss": "https://login.example.com/oauth2/default",
  "aud": "your-audience",
  "permissions": ["model-access-openai", "model-access-anthropic"]
}

Or equivalently:

{
  "permissions": "model-access-openai model-access-anthropic"
}

The group names must match the groups assigned to models in Middleman (see Model Groups below).

No permissions claim?

If the JWT carries no honoured permission — no permissions or cognito:groups claim and no allowlisted scp value — Hawk falls back to hawk:defaultPermissions (default: model-access-public). A scp claim holding only non-public scopes is filtered to nothing and counts as no permission, so it triggers this fallback too. This is how ungrouped Cognito users get access without custom claims.

Existing deployments upgrading in August 2026

When you upgrade an existing deployment to this release, model-access-public becomes an ordinary group — a user only sees public-model eval/scan data if they hold it. There are two ways a user can hold it, and most deployments are already covered:

From default permissions — hawk:defaultPermissions defaults to model-access-public and is applied to any user whose token carries no permissions/scp/cognito:groups claim. So claimless users (e.g. ungrouped Cognito users) keep access automatically, no action needed.
From their own claim — users whose tokens do carry explicit model-access-* claims must include model-access-public among them. The default is not merged into existing claims, so add it in your IdP for anyone who should see public data.

The only users who lose access are those whose explicit claims omit model-access-public — grant it to them.

Setting Up Your Identity Provider¶

The exact steps vary by provider, but the general approach is:

Create an OIDC application in your IdP (Okta, Auth0, Entra ID, etc.) with PKCE and the redirect URIs above.
Create groups in your IdP for each model access level you need (e.g., model-access-openai, model-access-anthropic). These must match the group names you assign to models in Middleman.
Add a custom claim named permissions to your access tokens that includes the user's group memberships. Most IdPs support this via:
- Okta: Custom claim on an authorization server using a Groups expression
- Auth0: Post-login Action that adds groups to the access token
- Entra ID (Azure AD): App roles or group claims in the token configuration
- Keycloak: Protocol mapper for group membership
Configure Hawk with your IdP's client ID, audience, and issuer URL in your Pulumi stack config.

Access Control¶

Model Groups¶

Each model configured in Middleman belongs to a model group. Permissions separate access to stored results from using the model:

Permission	Read/download results	Import/edit results	Call models
`model-access-example:read`	Yes	No	No
`model-access-example:write`	Yes	Yes	No
`model-access-example:execute`	No	No	Yes
`model-access-example`	Yes	Yes	Yes
No matching grant	No	No	No

Set these grants in the identity provider's trusted permissions or cognito:groups claim. Model registry and result-policy group names remain unsuffixed. Permissions are checked per group; access to one model group does not grant access to another.

Grants combine: :read plus :execute permits running evals and reading results, without editing results. :execute alone permits model calls, but cannot launch Hawk runs or read saved results for that group. Unsupported suffixes grant nothing.

Scans require read access to their source results and read plus execute access to their scanner models. Their results retain both groups' restrictions. Existing ownership requirements are listed in the endpoint audit.

Warehouse evaluation results are restricted by the models declared or observed in the evaluation. Folder and pod reads use their own model-group checks, as described in the endpoint audit.

Result access also controls visibility of usage, including user/team/job/task labels and estimated spend. Hawk admins can normally see usage for all models, but prices remain restricted to their readable model groups. With hawk:publicModelsOnly, usage is limited to public models even for admins.

For lockdown deployments, Middleman's authenticated /public_models_info endpoint exposes public-model metadata independently of the caller's groups, with secret details hidden. It does not grant permission to run those models.

Models are assigned to groups by Middleman admins when configuring the model. For example, invoking a model with group: "model-access-openai" requires model-access-openai or model-access-openai:execute in the user's JWT permissions claim. The model-access-public group is the default for models intended for all users.

How a model in a log is matched to its group¶

The warehouse records the canonical model name — the provider/lab prefix is stripped when the log is imported, so openrouter/z-ai/glm-5.2 is stored as glm-5.2. Middleman knows the model by the public_name it was registered under, which usually keeps that prefix. Group lookup therefore matches the two spellings against each other, accepting one name as a path suffix of the other, and a model registered either way gates its eval and scan data.

A suffix has to start at a / boundary on both sides: claude-3 matches anthropic/claude-3, but secret is a different model from anthropic/claude-secret. The importer follows the same rule when it picks a name, so the name it stores is always a whole-segment suffix of the registered one.

Upgrading from a release before this matching existed

Hawk previously compared the two names for exact equality. A model registered with a prefix never matched what the warehouse stored, so its evals and scans came out with no model group — and a resource with no model group is unrestricted. Those rows were readable by everyone.

The migration recomputes the cached groups, so on upgrade they become restricted to the model's group. Users who need them must hold that group; grant it before upgrading if they don't. Nothing is deleted — granting the group restores visibility.

Downgrading that migration restores the old matching for anything imported afterwards, but deliberately leaves the recomputed groups in place — rolling back must not re-publish the rows the upgrade restricted. To deliberately return those rows to their old, unrestricted state, clear their model_groups yourself after downgrading.

Declared and observed model dependencies¶

Warehouse restrictions include registered primary models, model roles and sample model calls, plus retained groups from import-time discovery. Unknown names add no restriction to the warehouse's model lookup. Empty groups pass warehouse, folder, pod and monitoring checks; a missing folder permission file is denied.

Running evals and pending, importing or failed imports retain their launch-time groups until terminal import succeeds. Scan reads also check current source-eval restrictions. Missing source transcripts carry __unresolved__, which denies ordinary warehouse access and cannot be granted; database administrators retain their RLS bypass.

Endpoint audit¶

Surface	Model permission	Ownership
`/meta/eval-sets`	All sibling evals, including those hidden by filters or row-level security	None
`/meta/jobs`	Launch groups and all imported results	None
`/meta/evals`, `/meta/scans`, sample metadata/scores/events/timeline and warehouse search	Warehouse groups; scans also check current source evals	None
Config, sample content, viewer logs/downloads, scan exports, transcript grep, eval-set status/stream and human-eval metadata	Folder groups; batch downloads check every folder	None
LLM transcript search	Read access to the transcript and execute access to the search model; cross-provider disclosure checks apply	None
Runner logs, traces and monitoring	Job/pod groups	None
Eval stop/delete; scan delete	No separate content check	Recorded owner or admin
Eval/human relaunch, including config-pinned IDs	Read and execute access to every recorded folder group; validate requested models before launch	Owner or admin
Scan resume	Read access to existing results and sources; read and execute access to current scanner groups	Owner or admin; active or unverified releases are refused
Fresh eval, human-eval and scan launches	Read and execute access to requested models; scans also require source read access	Reserve owner before launch writes; retain after failure
Eval/scan imports	Write access to all resulting model groups and existing content on updates	Reserve owner at allocation; owner/admin for uploads and finalization
Sample score overwrites and invalidation changes	Write access to folder groups	Run/import owner or admin
Append info/score events	Write access to folder groups	None
Relay, ACP, attach and browser approvals (`/approvals/{job_id}`, HTTP preflight and WebSocket)	Full access (read, write and execute) to every relevant folder/pod group and a write-capable token	None

Reading does not require ownership. Warehouse checks do not update folder permission files or pod groups, so access can differ between these surfaces. Viewer file writes/deletes are disabled; unsupported scan-viewer data paths return 400.

Pinned IDs: IDs with warehouse results or human assignments but empty S3 folders are refused, including for admins.
Legacy ownership: Native runs without a launch record require an admin with content access for resume or destructive edits; imports without an owner record require the same for updates or destructive edits. An admin resuming a legacy native run becomes its recorded owner; stop/delete still require a launch record.
Imports: Owner records live in the API-only import-owners/ prefix. Admins cannot finalize another uploader's stamped files without reuploading with their own attribution.
Failed scan creation: If config or permission files are missing, resubmit with hawk scan run to get a new run ID. The retained owner record alone does not make a scan resumable.

API-only helpers use the reserved hawk_api database role, granted to the API login (the first warehouseReadWriteUsers entry). Warehouse readers retain the scan helpers needed for row-level security.

Models discovered during import¶

New imports retain original model identities from the primary model, roles, eval and sample usage, and model events as work commits. On completion, the importer adds their resolved groups to warehouse restrictions and the existing .models.json. Both retain earlier restrictions; replacing content never automatically declassifies it. Native and imported scans inherit their settled sources' groups plus scanner groups.

Discovery adds no sample reconstruction pass on retry: previously committed names are read from the eval row. Existing header normalization reads remain unchanged. Rows with observed_model_names IS NULL are legacy imports and receive no automatic repair; an accepted forced full reimport opts them in. No historical backfill runs.

This is an eventual backstop. Undeclared calls may expose data before completion, so declare models before launch. Unknown models or missing/malformed policies fail publication through the existing importer retry path; they require operator attention if retries exhaust. Concurrent scans and source writes receive no additional guarantee. See the deployment procedure.

Provider-discovered models¶

When Middleman loads its database-backed registry, it also records models found in provider feeds and their model-access groups. Stored names and their original groups are retained even if a provider stops listing them. The warehouse resolves these names with the same suffix matching as administrator-registered models.

Discovery adds missing restrictions to matching existing evals and scans, including scans that read those evals. It preserves every cached group and the __unresolved__ marker that hides unfinished scan imports. Registry reloads retry stored names, so an import that overlaps the first discovery is repaired on the next successful load. A database failure is logged and does not stop Middleman from serving its registry; repair waits for a successful retry.

This backfill targets full names and their path suffixes in eval, model-role, sample-model, and scan records. It does not search sample usage JSON or the reverse suffix direction; those cases wait for their normal import refresh. Discovery does not rebuild jobs or .models.json, and changing an existing model's group still requires deliberately rebuilding the affected caches.

Read-only scopes¶

Credentials whose model grants are all :read cannot make mutating requests (403). Use these for machine or agent tokens that only need to read results.

How Group Membership Flows¶

flowchart LR
    IdP["Identity Provider<br/>(Okta, Auth0, Cognito)"]
    JWT["JWT with<br/>permissions claim"]
    API["Hawk API"]
    MM["Middleman"]
    TB["Token Broker<br/>(Lambda)"]
    S3["S3 (eval logs)"]
    DB["PostgreSQL<br/>(RLS)"]

    IdP -->|"issues"| JWT
    JWT -->|"validates"| API
    JWT -->|"validates"| MM
    JWT -->|"exchanges for<br/>scoped credentials"| TB
    TB -->|"scoped access"| S3
    API -->|"RLS enforces<br/>group membership"| DB

Identity Provider issues JWTs with model group memberships in the permissions claim
Middleman validates the JWT and checks the user's groups before routing model API calls
Token Broker validates the user's model group permissions, then exchanges the JWT for scoped AWS credentials tied to a specific job via AWS session tags
PostgreSQL RLS (Row-Level Security) restricts database queries to evaluation results the user is authorized to see

Token Broker Job Identity¶

The broker always validates the user's access token. Runners also send a projected Kubernetes ServiceAccount identity token so the broker can bind the request to a specific job:

Factor	Header	Proves	Issued by
User access token	`Authorization: Bearer`	Permission — the caller's model grants authorize the requested operation	Your OIDC provider
Job identity token	`X-Hawk-Job-Token`	Identity — the caller really is the job whose `job_id` it requests	The EKS cluster's OIDC issuer

The job identity token is a Kubernetes projected ServiceAccount token mounted into the runner pod, scoped to the hawk-token-broker audience and rotated by kubelet. The broker validates its signature against the cluster's JWKS endpoint, then checks that the token's sub equals the full system:serviceaccount:<runner-namespace>:<runner-service-account> it derives for the requested job, and that the token carries a pod binding.

With hawk:requireJobToken: "true", both tokens are required and the user token alone is insufficient. This prevents a runner with broad model permissions from requesting credentials session-tagged for another eval set's job_id and gaining access to that eval set's S3 prefix. The default is currently permissive, as described below, so deployments do not get this two-factor guarantee until they explicitly enable enforcement.

Output credentials without verified worker identity require write and execute access to every result group. Workers without those grants need a matching, pod-bound job token even when requireJobToken is false.

Enforcing (`requireJobToken`)¶

hawk:requireJobToken is false (permissive) by default:

Permissive — a missing, invalid, or mismatched job token is logged as an audit warning and emits a JobIdentitySkipped CloudWatch metric, dimensioned by error_type (missing, invalid, mismatch, unbound, error). Credentials may still be issued if the caller independently satisfies the required model permissions above.
Enforce (true) — the same conditions return 403 and emit JobIdentityDenied with the same dimensions.

Enforcing kills in-flight jobs launched before the chart change

Only runners created after the Helm chart change have the projected token volume. Flipping requireJobToken to true makes every older runner fail at its next credential refresh — which, because credentials last an hour, can be up to an hour after the flip and appears as a mid-eval S3 failure.

Rollout:

Deploy with requireJobToken: false and let the existing jobs drain.
Watch the "Token broker: job identity (permissive skips)" widget on the Hawk platform overview dashboard. missing decays toward zero as pre-rollout jobs finish; it should stay at zero for jobs launched after the deploy. A persistent non-zero error usually means the broker's JOB_TOKEN_* configuration is wrong rather than that a runner misbehaved.
Once no newly launched job reports missing, and no job is still running from before the chart change, set requireJobToken: true.

mismatch is the one reason that is never benign — it means a caller asked for a job it does not hold the identity token for. In permissive mode credentials can still be issued to callers with sufficient direct permissions. When hawk:enableProdAlarms is true, the matching <env>-hawk-token-broker-identity-mismatch-permissive or <env>-hawk-token-broker-identity-mismatch-denied alarm fires on the first occurrence. Subscribe a receiver to their shared SNS topic before starting the rollout. The flag controls alarm creation regardless of the stack name; alarms are not created merely because a stack is named prd.

In enforce mode the broker refuses to start a request with empty JOB_TOKEN_* configuration, and pulumi up fails when requireJobToken is enabled without a resolvable EKS OIDC provider URL — an empty issuer would otherwise 403 every runner while looking correctly configured.

Administrative Roles¶

Hawk has one administrative role: Middleman Admin. Admins can:

Create, update, and delete model configurations
View all models regardless of group membership
Manage provider API keys

To grant admin access, add a boolean claim to your OIDC access tokens:

Claim	Purpose
`https://middleman.metr.org/claims/admin`	Full admin access (production + non-production)
`https://middleman.metr.org/claims/dev-admin`	Admin access for non-production environments only

The dev-admin claim is only accepted when the Middleman environment variable MIDDLEMAN_ACCEPT_DEV_ADMIN=true is set (which Hawk configures automatically for non-production stacks).

Example JWT with admin access:

{
  "sub": "admin-user",
  "permissions": ["model-access-openai"],
  "https://middleman.metr.org/claims/admin": true
}

In your IdP, create a group (e.g., middleman-admins) and configure a custom claim that emits true when the user is a member of that group.

Hawk Admin¶

Hawk admins can stop, delete and resume other users' runs, update imports and sample scores/invalidation, manage human-registry entries, and author code-access rules. Admin status bypasses ownership checks, not model access. The endpoint audit lists the checks and legacy exceptions.

There are two OR-ed sources of admin status; both are disabled by default:

Config key	How it works
`hawk:hawkAdminClaim`	Name of a JWT claim. When the claim is present and its value is boolean `true`, the bearer is treated as a Hawk admin.
`hawk:hawkAdminPermissions`	List of permission/group names. If any entry matches a value in the token's `permissions` or `cognito:groups` claim, the bearer is treated as a Hawk admin. Values from `scp` cannot confer admin — only public model-access scopes are honoured from `scp`, and those may not be admin names.

hawkAdminPermissions entries must be disjoint from defaultPermissions: tokens that carry no permission claims have the defaults substituted in, so any overlap would make every such caller an admin. The API refuses to start if it detects an overlap. The entries must also not be OAuth scopes that users can self-request from your IdP.

Ownership overrides emit a warning-level admin_override event with the action, actor and recorded owner (null when unavailable). Job events include job_id; human-registry events include human. Actions are stop, delete, resume, edit, update_import or update.

Browser approval WebSockets validate the API session cookie/JWT and viewer Origin before upgrading, and close when the credential expires. Content access and write capability are rechecked before sending each decision. approval_decision logs record the actor, run, session, tool call, offered choice and delivery or application status; tool arguments and credentials are not included. Browser approval sessions do not expose general ACP commands or arbitrary relay ports. These audit records identify an account, not proof of human intervention.

Browser approval review requires an HTTPS/WSS relay outside localhost because the API forwards the reviewer's bearer token to the relay. HTTP-only testing deployments using hawk:skipTlsCerts: "true" must enable TLS to use this feature.

The adapter parses runner-controlled ACP messages, capped at 2 MiB each. A review retains at most 10,000 samples and 64 pending approvals, with a 4 MiB total limit on serialized pending approval data. Exceeding a limit closes the review without answering pending requests; use hawk acp for runs exceeding these limits.

Runner pods receive the launching user's access token. An admin's token lets task code exercise these same powers—including resume, import updates and destructive sample edits—while model-access checks still apply. Prefer separate day-to-day and admin identities, or short-lived admin membership.

Example: using hawk:hawkAdminClaim with an Okta custom claim:

{
  "sub": "admin-user",
  "permissions": ["model-access-openai"],
  "hawk-admin": true
}

pulumi config set hawk:hawkAdminClaim hawk-admin

For Cognito deployments, hawk:hawkAdminPermissions is simpler — create a Cognito group (e.g. hawk-admin) and add it to the config:

pulumi config set --path 'hawk:hawkAdminPermissions[0]' hawk-admin

Code-access rule authoring¶

Administrators can create and edit code-access policy through /code_rules/. This is policy authoring only; it does not yet restrict submissions, results, or artifacts. See Code-access rules for the API, matching semantics, and the remaining enforcement work.

Sensitive Model Protection¶

Hawk protects sensitive model information through codenames:

Models are given a public_name (codename) that is used everywhere in the system
The real model identifier (danger_name) is only known to Middleman
Evaluation results, logs, and the web viewer only show the public_name
Observability telemetry uses layered protections: model fields and tags use public_name, known credential fields are removed, and recognized model-bearing URL forms are scrubbed. URL and arbitrary-text scrubbing are best-effort, so observability access must remain restricted.

Users without the appropriate model group cannot access evaluation results that used that model.

For models with are_details_secret, the usage price endpoint also hides prices and provider/lab details.

Sandbox Isolation¶

Evaluations run in isolated Kubernetes pods with:

Separate namespaces — each evaluation gets its own namespace for the runner and sandbox pods
Network policies — Cilium network policies block egress to VPC infrastructure (primary subnet CIDR, EC2 IMDS, EKS Pod Identity) from sandbox pods, and can restrict no-internet pods to DNS-only egress
Resource limits — CPU and memory constraints per pod
StatefulSets — sandbox pods auto-restart on failure

Sandbox egress is deny-by-default; internet access is opt-in per task. For how to configure it per evaluation (and the main use case, cyber-related evals), see Sandbox Network Isolation.

Alternative Sandbox Providers¶

While Kubernetes is the default sandbox environment, Hawk's architecture does not strictly require it. EC2-based sandboxing and other providers (e.g., Modal) can be used as alternatives. The sandbox provider is configured per evaluation.

gVisor RuntimeClass and CNI mode¶

Set hawk:enableGvisor: "true" to install the gvisor RuntimeClass. Pods opt into gVisor with runtimeClassName: gvisor; the RuntimeClass uses the pinned runsc runtime. Enabling it changes the default Karpenter node class from Bottlerocket to AL2023 so the runtime can be installed during node bootstrap. The AL2023 node configuration preserves single-process OOM killing for workload containers.

The default hawk:ciliumExclusive: "false" keeps Cilium chained to AWS VPC CNI and retains the default RFC 1918 Cilium pod pool. Exclusive mode makes Cilium the node CNI, uses an RFC 1918 overlay pod pool, and retires the aws-node DaemonSet after Cilium is ready. Treat that as a CNI migration for existing clusters: plan node recycling and a rollback path before enabling it.

Audit Logging¶

Application-Level Logging¶

Hawk API — all API requests are logged to CloudWatch with user identity, action, and resource context
Middleman — at the default summary traffic-log level, model API calls are logged with user identity, public model name, token usage, and body sizes, but request/response bodies are not retained. At hawk:middlemanTrafficLogLevel: full, bounded raw bodies are archived in S3: known headers and the exact top-level api_key field in a parseable JSON request object are redacted, but other request values and response bodies are not value-scrubbed. Treat this archive as sensitive.
Token Broker — credential exchanges are logged with user identity and requested scope

AWS CloudTrail¶

Hawk does not provision a CloudTrail trail. Configure organization- or account-level trails, destinations, retention, and optional CloudTrail Insights outside this repository. In METR's deployment, AWS Control Tower supplies the organization trail and its existing destinations; METR's private infra-shared repository adopts that trail to configure Insights and selected data events and adds query and monitoring integration. Infra-shared does not create the Control Tower log bucket/log group or control their retention.

VPC Flow Logs¶

Hawk does not provision VPC Flow Logs. If you need network-level audit logging, enable VPC Flow Logs separately and choose a CloudWatch Logs or S3 destination and retention policy that fits your deployment. hawk:cloudwatchLogsRetentionDays (default: 14 days) does not control a separately managed flow-log group. METR's infra-shared deployment enables all-traffic flow logs for staging and production at /aws/vpc/flowlogs/<env> with 365-day CloudWatch retention; production also uses an S3 destination.

Endpoint Protection (CrowdStrike Falcon)¶

Hawk optionally deploys the CrowdStrike Falcon sensor to protect infrastructure hosts. Enable it with hawk:enableCrowdstrike: "true" and a Secrets Manager secret containing your CrowdStrike API credentials (see Configuration).

What's Protected¶

Target	OS / Arch	Installation Method
All EKS nodes (Karpenter)	Bottlerocket and AL2023	DaemonSet via falcon-sensor Helm chart (requires Falcon Images Download scope)
Tailscale subnet router	AL2023 / ARM64	Sensor RPM installed via cloud-init at boot

EKS nodes get the sensor from the DaemonSet and only from the DaemonSet. GPU nodes once also installed it as a host RPM from EC2NodeClass userData; they no longer do, because a node cannot run both — the containerized sensor exits immediately on a host that already owns /opt/CrowdStrike, leaving its pod in CrashLoopBackOff. Hosts outside EKS have no DaemonSet to rely on and still install the RPM.

The DaemonSet pulls a container image from registry.crowdstrike.com, which requires the Falcon Images Download: Read scope — part of the Falcon Cloud Security with Containers add-on. The subnet router downloads its RPM from the CrowdStrike API at boot using the Sensor Download: Read scope.

Tailscale ZTA Integration¶

With the Falcon sensor on the subnet router, you can enable CrowdStrike ZTA with Tailscale to gate tailnet access based on device posture. This requires a separate API client with Hosts: Read and Zero Trust: Read scopes, configured in the Tailscale admin console under Device Posture Integrations. Note that ZTA scores apply to the router instance itself, not to traffic routed through it.

Runtime feature flags¶

Operators can change runtime flags without a merge, deployment, or API restart. Each deployment gets an AWS AppConfig feature-flags profile from Pulumi: application <env>-hawk-api-feature-flags, environment <env>, configuration profile feature-flags. The API reads the deployed configuration on demand and caches it for ten seconds, so a toggle reaches every replica within about that long. Requests that already read the flag finish under the previous value, and changing a flag does not stop running jobs.

How the cross-lab rule decides¶

Hawk stops a model from reading non-public transcripts produced by models that another company serves. The check runs for standalone scans (create and resume), online scans in eval sets (create and resume), and LLM transcript search. The receiving model is the scanner or search model; the source models are the models whose transcripts it would read.

Both sides are judged by the serving provider that Middleman records for the model in its /model_groups response. The provider is the company that receives the data. A model served through OpenRouter or another aggregator has that aggregator as its provider, since sending it data hands that data to the aggregator. Middleman's per-integration lab is not used, so two models served by the same company through different integrations may read each other's transcripts. Providers must match exactly, with no normalization.

Models are looked up by their Middleman public names. For scans, the inspect provider prefix in a config (for example openai-api/openai-chat/...) is not used, because it only picks the wire format, while Middleman decides where the request goes. Transcript search looks up the search model and the sample's models by their Middleman public names, taken from the sample's recorded model calls. For older data without them, it falls back to the names stored in the warehouse.

Public source models are exempt and need no provider, and a scan with no receiving model (one that uses only non-LLM scanners) is not checked. A model with no recorded provider gets its lab's default from Middleman, so only models on labs without one (the dummy test labs) have none. If the receiving model or any non-public source model has no provider, the request is refused with a 422 that names those models. The cross-lab override does not apply to this error; an administrator has to record the missing provider in Middleman. A deployment therefore needs a Middleman that returns providers from /model_groups.

Cross-lab scan overrides¶

allow_sensitive_cross_lab_scan_override controls whether the API honors the existing allow_sensitive_cross_lab_scan request field (hawk scan run/resume --allow-sensitive-cross-lab-scan). It applies to standalone scan creation, scan resume, and eval creation/resume with an online scan.

Runtime flag	Request asks for override	Behavior for non-public cross-lab transcripts
Enabled (default)	No	Blocked with 403
Enabled (default)	Yes	Allowed, subject to normal model and transcript permissions
Disabled	Either	Blocked with 403

If the request asks for an override and the flag store cannot be read, the request fails with a 503 instead; see below.

Same-provider scans and public source models keep their existing behavior. This flag does not affect LLM transcript search or change how providers compare. It covers only provider mismatches: a request refused because a provider is missing stays refused with a 422 whatever the flag says.

Toggling a flag¶

In the AWS console: AWS AppConfig → Applications → <env>-hawk-api-feature-flags → Configuration profiles → feature-flags, flip the flag, Save new version, then Start deployment into the <env> environment with the AppConfig.AllAtOnce strategy. The same two steps from the CLI, using the IDs shown in the console:

cat > flags.json <<'JSON'
{"version": "1",
 "flags": {"allow_sensitive_cross_lab_scan_override": {"name": "allow_sensitive_cross_lab_scan_override"}},
 "values": {"allow_sensitive_cross_lab_scan_override": {"enabled": true}}}
JSON
configuration_version=$(aws appconfig create-hosted-configuration-version --application-id <app-id> \
  --configuration-profile-id <profile-id> --content-type application/json \
  --content fileb://flags.json --query VersionNumber --output text /dev/null)
aws appconfig start-deployment --application-id <app-id> --environment-id <env-id> \
  --configuration-profile-id <profile-id> --deployment-strategy-id AppConfig.AllAtOnce \
  --configuration-version "$configuration_version"

Deployments use the built-in AppConfig.AllAtOnce strategy: the new value is served immediately, then the environment bakes for ten minutes, during which StartDeployment is rejected with ConflictException. To reverse a toggle inside that window, stop the in-flight deployment instead. That rolls the environment back to the previous version, which means every flag in the profile, not just the one you changed:

aws appconfig stop-deployment --application-id <app-id> --environment-id <env-id> \
  --deployment-number "$(aws appconfig list-deployments --application-id <app-id> \
    --environment-id <env-id> --query 'Items[0].DeploymentNumber' --output text)"

Right after Pulumi seeds the profile (the stack's first deploy) there is no earlier version to roll back to, so the first toggle may have to wait out that bake. The bake is kept deliberately: it is what lets AppConfig roll back automatically if CloudWatch alarms are ever attached to the environment.

Every flag declares a default in the hawk_flags package: the value in effect until an operator changes it. A flag that is absent from the deployed configuration, or whose enabled value is not a JSON boolean, uses that default, as does a deployment with no HAWK_API_FEATURE_FLAGS_* settings. If AppConfig cannot be read (access denied, throttling, nothing deployed yet), a request that asked for an override fails with a 503 and should be retried. Requests without an override, and requests with no cross-lab violation, never read the flag.

Pulumi seeds the profile with every flag at its default when its AppConfig resources are first created. Later updates ignore changes to the seed content and the seed deployment's version, strategy, description and encryption key, so those edits do not replay the defaults over operator toggles. A flag added to Hawk later is absent from an existing environment's profile and runs at its default there; to toggle it, add it to the profile in the console first. Changing a flag's default in code likewise reaches only new profiles and environments where the flag is absent: wherever an operator's version carries the flag, its deployed value stands.

This protection does not cover forced replacement or recreation after losing Pulumi state. Replacing the AppConfig application, environment or profile also initializes the new target with the defaults. Preserve the current flag values before these operations and restore them afterwards if needed.

Upgrade behavior: the flag defaults to enabled, so upgrading changes nothing for callers. To turn the override off, deploy a disabled version once the AppConfig profile exists (after the first deployment of this version). Until every API replica has upgraded, older replicas still honor per-request overrides independently of this flag, so a disable takes full effect only once the rollout completes.

AWS Security Services¶

In METR's deployment, AWS security services (GuardDuty, Security Hub, AWS Config, and CloudTrail Insights) are managed by METR's private infra-shared repository, not by Hawk. Other Hawk operators must configure equivalent account- or organization-level services separately.

Recommended Security Configuration¶

For production deployments, consider:

Setting hawk:eksPublicEndpoint: "false" and using Tailscale for private cluster access. If the public EKS API endpoint must remain enabled, set hawk:eksPublicAccessCidrs to restrict it to approved source CIDRs; omitting it preserves AWS's 0.0.0.0/0 default.
Setting hawk:albInternal: "true" to make the ALB private (requires VPN)
Setting hawk:protectResources: "true" to prevent accidental deletion of stateful resources (S3 buckets, secrets, the Datadog log-archive bucket, and the Aurora cluster)

Monitoring & Observability¶

CloudWatch¶

Hawk's application and service logs go to CloudWatch by default. The Hawk-owned groups configured with hawk:cloudwatchLogsRetentionDays retain 14 days by default.

Key Hawk-owned log groups include:

Hawk API logs
Middleman logs
Lambda function logs

In METR's deployment, infra-shared separately owns the GuardDuty and Security Hub finding groups at /aws/events/guardduty/<env> and /aws/events/securityhub/<env>. They retain 365 days and are not controlled by hawk:cloudwatchLogsRetentionDays.

Datadog (Optional)¶

For richer monitoring, Hawk supports Datadog integration:

hawk:enableDatadog: "true"

This enables:

APM — distributed tracing across API, Middleman, and Lambda functions
Log forwarding — CloudWatch logs forwarded to Datadog
Custom metrics — token usage, import counts, evaluation durations
Sensitive data filtering — known sensitive fields and auth headers are removed, while model fields and tags use public names. Model-bearing URL scrubbing is best-effort for recognized provider forms; arbitrary text and unrecognized URLs are not guaranteed to be value-scrubbed. The optional full Middleman traffic archive follows the narrower body-redaction rules described above.

Budget Alerts¶

Monitor AWS spending with budget alerts:

hawk:budgetLimit: "10000"
hawk:budgetNotificationEmails:
  - "team@example.com"
hawk:budgetNotificationThresholds:
  - 80
  - 100

External Dependencies¶

Hawk interacts with the following external services:

Service	Purpose	Required?
Docker Hub	Pull base container images	Yes (login recommended to avoid rate limits)
LLM Providers	Model API calls via Middleman	At least one provider API key required
OIDC Provider	User authentication	Optional (Cognito used by default)
Datadog	Monitoring and observability	Optional
Slack	Budget alerts	Optional
CrowdStrike Falcon	Endpoint protection for EKS nodes and subnet router	Optional
Tailscale	Private VPC, EKS API, and internal ALB access	Optional
Cloudflare	DNS delegation	Optional
GitHub	CI/CD via Pulumi Deploy	Optional

Network Security¶

TLS by default — external service traffic uses ACM certificates unless an operator explicitly enables the testing-only hawk:skipTlsCerts HTTP mode
Private subnets — EKS nodes, RDS, and ECS tasks run in private subnets with no direct internet access
NAT gateway — the default VPC routes outbound internet access from all private subnets through one shared NAT gateway
Security groups — restrict traffic between components (ALB → ECS, ECS → RDS, etc.)
VPC endpoints — S3 traffic stays within the VPC via a Gateway endpoint
Cilium network policies — pod-level network isolation within Kubernetes

Code source inventory¶

Native launches record declared package sources before Helm. Parsing gaps leave incomplete inventory without blocking launch; database recording failures stop preparation. Ordinary resumes retain earlier sources. An interrupted first launch may require a new ID, and incomplete inventory requires audit before enforcement. Inventory does not enforce code permissions or prove which code a runner installs.