Security¶

This page covers Hawk's security architecture, access control, audit logging, and optional AWS security services.

Authentication¶

Hawk uses OIDC (OpenID Connect) for all authentication. JWTs are validated at every service boundary — the API server, Middleman (LLM proxy), and Lambda functions. The web viewer is a static single-page app: it performs the OIDC login in the browser, and every data request it makes is validated by the API.

Default: Cognito¶

When no OIDC provider is configured, Hawk creates a Cognito user pool automatically. Users are managed via the AWS Console or the helper scripts.

Managing Users¶

scripts/dev/create-cognito-user.sh <stack> user@example.com

Managing Model Access Groups¶

Cognito automatically includes group memberships in the cognito:groups claim of access tokens. Create groups matching the model groups configured in Middleman:

# Create a model access group
scripts/dev/manage-cognito-groups.sh <stack> create model-access-openai

# Add a user to the group
scripts/dev/manage-cognito-groups.sh <stack> add-user model-access-openai user@example.com

# List all groups
scripts/dev/manage-cognito-groups.sh <stack> list

Users who aren't in any group fall back to hawk:defaultPermissions (default: model-access-public), which grants access to models that belong to the group model-access-public.

External OIDC Provider (Okta, Auth0, etc.)¶

For production deployments, we recommend using your organization's identity provider. Use the autodiscovery script to generate Pulumi config from your issuer URL:

uv run python scripts/dev/discover-oidc.py <your-issuer-url> <your-client-id> <your-audience>

This prints the full set of hawk:oidc* config values to add to your Pulumi.<stack>.yaml. See Pulumi.example.yaml for the complete list of OIDC settings.

OIDC App Requirements¶

Your OIDC application must:

Use PKCE (Proof Key for Code Exchange) — no client secret
Support the authorization_code grant type
Include these redirect URIs:
- http://localhost:18922/callback — Hawk CLI login
- https://viewer.<your-domain>/oauth/complete — web viewer login

Required JWT Claims¶

Hawk extracts permissions from the permissions claim (or scp as a fallback). The claim value can be either a JSON array of strings or a space-separated string:

{
  "sub": "user123",
  "iss": "https://login.example.com/oauth2/default",
  "aud": "your-audience",
  "permissions": ["model-access-openai", "model-access-anthropic"]
}

Or equivalently:

{
  "permissions": "model-access-openai model-access-anthropic"
}

The group names must match the groups assigned to models in Middleman (see Model Groups below).

No permissions claim?

If the JWT has no permissions, scp, or cognito:groups claim, Hawk falls back to hawk:defaultPermissions (default: model-access-public). This is how ungrouped Cognito users get access without custom claims.

Existing deployments upgrading in August 2026

When you upgrade an existing deployment to this release, model-access-public becomes an ordinary group — a user only sees public-model eval/scan data if they hold it. There are two ways a user can hold it, and most deployments are already covered:

From default permissions — hawk:defaultPermissions defaults to model-access-public and is applied to any user whose token carries no permissions/scp/cognito:groups claim. So claimless users (e.g. ungrouped Cognito users) keep access automatically, no action needed.
From their own claim — users whose tokens do carry explicit model-access-* claims must include model-access-public among them. The default is not merged into existing claims, so add it in your IdP for anyone who should see public data.

The only users who lose access are those whose explicit claims omit model-access-public — grant it to them.

Setting Up Your Identity Provider¶

The exact steps vary by provider, but the general approach is:

Create an OIDC application in your IdP (Okta, Auth0, Entra ID, etc.) with PKCE and the redirect URIs above.
Create groups in your IdP for each model access level you need (e.g., model-access-openai, model-access-anthropic). These must match the group names you assign to models in Middleman.
Add a custom claim named permissions to your access tokens that includes the user's group memberships. Most IdPs support this via:
- Okta: Custom claim on an authorization server using a Groups expression
- Auth0: Post-login Action that adds groups to the access token
- Entra ID (Azure AD): App roles or group claims in the token configuration
- Keycloak: Protocol mapper for group membership
Configure Hawk with your IdP's client ID, audience, and issuer URL in your Pulumi stack config.

Access Control¶

Model Groups¶

Model access is controlled through model groups. Each model configured in Middleman belongs to a group (e.g., model-access-openai, model-access-anthropic). Users must have a matching group in their JWT permissions claim to:

Use the model for evaluations (via Middleman)
View evaluation results that used that model (via the web viewer and API)

This means evaluation results are automatically restricted to users who have access to the models used in the evaluation.

Models are assigned to groups by Middleman admins when configuring the model. For example, a model with group: "model-access-openai" requires the user to have model-access-openai in their JWT permissions claim. The model-access-public group is the default and grants access to models intended for all users.

How a model in a log is matched to its group¶

The warehouse records the canonical model name — the provider/lab prefix is stripped when the log is imported, so openrouter/z-ai/glm-5.2 is stored as glm-5.2. Middleman knows the model by the public_name it was registered under, which usually keeps that prefix. Group lookup therefore matches the two spellings against each other, accepting one name as a path suffix of the other, and a model registered either way gates its eval and scan data.

A suffix has to start at a / boundary on both sides: claude-3 matches anthropic/claude-3, but secret is a different model from anthropic/claude-secret. The importer follows the same rule when it picks a name, so the name it stores is always a whole-segment suffix of the registered one.

Upgrading from a release before this matching existed

Hawk previously compared the two names for exact equality. A model registered with a prefix never matched what the warehouse stored, so its evals and scans came out with no model group — and a resource with no model group is unrestricted. Those rows were readable by everyone.

The migration recomputes the cached groups, so on upgrade they become restricted to the model's group. Users who need them must hold that group; grant it before upgrading if they don't. Nothing is deleted — granting the group restores visibility.

Downgrading that migration restores the old matching for anything imported afterwards, but deliberately leaves the recomputed groups in place — rolling back must not re-publish the rows the upgrade restricted. To deliberately return those rows to their old, unrestricted state, clear their model_groups yourself after downgrading.

Read-only scopes¶

A model-access permission with a :read suffix (e.g. model-access-public:read) grants read-only access: the principal can browse eval data gated by that group but cannot submit eval sets, run scans, import logs, or edit samples — mutating requests are rejected with 403. Use this for machine/agent tokens that only need to read results. A plain scope (no suffix) grants both read and write.

How Group Membership Flows¶

flowchart LR
    IdP["Identity Provider<br/>(Okta, Auth0, Cognito)"]
    JWT["JWT with<br/>permissions claim"]
    API["Hawk API"]
    MM["Middleman"]
    TB["Token Broker<br/>(Lambda)"]
    S3["S3 (eval logs)"]
    DB["PostgreSQL<br/>(RLS)"]

    IdP -->|"issues"| JWT
    JWT -->|"validates"| API
    JWT -->|"validates"| MM
    JWT -->|"exchanges for<br/>scoped credentials"| TB
    TB -->|"scoped access"| S3
    API -->|"RLS enforces<br/>group membership"| DB

Identity Provider issues JWTs with model group memberships in the permissions claim
Middleman validates the JWT and checks the user's groups before routing model API calls
Token Broker validates the user's model group permissions, then exchanges the JWT for scoped AWS credentials tied to a specific job via AWS session tags
PostgreSQL RLS (Row-Level Security) restricts database queries to evaluation results the user is authorized to see

Token Broker Job Identity¶

The broker requires two factors from two different subjects before it issues credentials for a job:

Factor	Header	Proves	Issued by
User access token	`Authorization: Bearer`	Permission — the caller's model groups allow reading this eval set	Your OIDC provider
Job identity token	`X-Hawk-Job-Token`	Identity — the caller really is the job whose `job_id` it requests	The EKS cluster's OIDC issuer

The job identity token is a Kubernetes projected ServiceAccount token mounted into the runner pod, scoped to the hawk-token-broker audience and rotated by kubelet. The broker validates its signature against the cluster's JWKS endpoint, then checks that the token's sub equals the full system:serviceaccount:<runner-namespace>:<runner-service-account> it derives for the requested job, and that the token carries a pod binding.

The user token alone is not sufficient: model-group read access is held by many jobs, so without the second factor any runner could request credentials session-tagged for another eval set's job_id and gain read/write/delete on that eval set's S3 prefix.

Enforcing (`requireJobToken`)¶

hawk:requireJobToken is false (permissive) by default:

Permissive — a missing, invalid, or mismatched job token is logged as an audit warning and emits a JobIdentitySkipped CloudWatch metric, dimensioned by error_type (missing, invalid, mismatch, unbound, error). Credentials are still issued.
Enforce (true) — the same conditions return 403 and emit JobIdentityDenied with the same dimensions.

Enforcing kills in-flight jobs launched before the chart change

Only runners created after the Helm chart change have the projected token volume. Flipping requireJobToken to true makes every older runner fail at its next credential refresh — which, because credentials last an hour, can be up to an hour after the flip and appears as a mid-eval S3 failure.

Rollout:

Deploy with requireJobToken: false and let the existing jobs drain.
Watch the "Token broker: job identity (permissive skips)" widget on the Hawk platform overview dashboard. missing decays toward zero as pre-rollout jobs finish; it should stay at zero for jobs launched after the deploy. A persistent non-zero error usually means the broker's JOB_TOKEN_* configuration is wrong rather than that a runner misbehaved.
Once no newly launched job reports missing, and no job is still running from before the chart change, set requireJobToken: true.

mismatch is the one reason that is never benign — it means a caller asked for a job it does not hold the identity token for. In permissive mode those credentials are still issued, so the <env>-hawk-token-broker-identity-mismatch alarm fires on the first occurrence in either mode. Subscribe a receiver to its SNS topic before starting the rollout; the alarm exists only in prd.

In enforce mode the broker refuses to start a request with empty JOB_TOKEN_* configuration, and pulumi up fails when requireJobToken is enabled without a resolvable EKS OIDC provider URL — an empty issuer would otherwise 403 every runner while looking correctly configured.

Administrative Roles¶

Hawk has one administrative role: Middleman Admin. Admins can:

Create, update, and delete model configurations
View all models regardless of group membership
Manage provider API keys

To grant admin access, add a boolean claim to your OIDC access tokens:

Claim	Purpose
`https://middleman.metr.org/claims/admin`	Full admin access (production + non-production)
`https://middleman.metr.org/claims/dev-admin`	Admin access for non-production environments only

The dev-admin claim is only accepted when the Middleman environment variable MIDDLEMAN_ACCEPT_DEV_ADMIN=true is set (which Hawk configures automatically for non-production stacks).

Example JWT with admin access:

{
  "sub": "admin-user",
  "permissions": ["model-access-openai"],
  "https://middleman.metr.org/claims/admin": true
}

In your IdP, create a group (e.g., middleman-admins) and configure a custom claim that emits true when the user is a member of that group.

Hawk Admin¶

Hawk admins can stop and delete eval sets and scan runs they do not own. Normal users are restricted to resources they created. Admin status has no effect on model or eval data access — viewing evaluation results remains gated by model-group membership as usual.

There are two OR-ed sources of admin status; both are disabled by default:

Config key	How it works
`hawk:hawkAdminClaim`	Name of a JWT claim. When the claim is present and its value is boolean `true`, the bearer is treated as a Hawk admin.
`hawk:hawkAdminPermissions`	List of permission/group names. If any entry matches a value in the token's `permissions`, `scp`, or `cognito:groups` claim, the bearer is treated as a Hawk admin.

hawkAdminPermissions entries must be disjoint from defaultPermissions: tokens that carry no permission claims have the defaults substituted in, so any overlap would make every such caller an admin. The API refuses to start if it detects an overlap. The entries must also not be OAuth scopes that users can self-request from your IdP.

Every admin override is logged at warning level as a structured admin_override event that includes the action (stop/delete), the actor identity, the resource owner, and the job ID, so all non-owner mutations are auditable.

Note that runner pods receive the launching user's access token (for model API and artifact access). When an admin launches an eval or scan, that delegated token carries admin status for its lifetime, so workloads in the run could use it to stop or delete other users' jobs. Prefer separate day-to-day and admin identities, or grant admin via short-lived group membership.

Example: using hawk:hawkAdminClaim with an Okta custom claim:

{
  "sub": "admin-user",
  "permissions": ["model-access-openai"],
  "hawk-admin": true
}

pulumi config set hawk:hawkAdminClaim hawk-admin

For Cognito deployments, hawk:hawkAdminPermissions is simpler — create a Cognito group (e.g. hawk-admin) and add it to the config:

pulumi config set --path 'hawk:hawkAdminPermissions[0]' hawk-admin

Sensitive Model Protection¶

Hawk protects sensitive model information through codenames:

Models are given a public_name (codename) that is used everywhere in the system
The real model identifier (danger_name) is only known to Middleman
Evaluation results, logs, and the web viewer only show the public_name
Observability data (Datadog, Sentry) is scrubbed to prevent danger_name leakage — API keys, auth headers, and model identifiers are filtered at multiple layers

Users without the appropriate model group cannot access evaluation results that used that model.

Sandbox Isolation¶

Evaluations run in isolated Kubernetes pods with:

Separate namespaces — each evaluation gets its own namespace for the runner and sandbox pods
Network policies — Cilium network policies block egress to VPC infrastructure (primary subnet CIDR, EC2 IMDS, EKS Pod Identity) from sandbox pods, and can restrict no-internet pods to DNS-only egress
Resource limits — CPU and memory constraints per pod
StatefulSets — sandbox pods auto-restart on failure

Sandbox egress is deny-by-default; internet access is opt-in per task. For how to configure it per evaluation (and the main use case, cyber-related evals), see Sandbox Network Isolation.

Alternative Sandbox Providers¶

While Kubernetes is the default sandbox environment, Hawk's architecture does not strictly require it. EC2-based sandboxing and other providers (e.g., Modal) can be used as alternatives. The sandbox provider is configured per evaluation.

gVisor RuntimeClass and CNI mode¶

Set hawk:enableGvisor: "true" to install the gvisor RuntimeClass. Pods opt into gVisor with runtimeClassName: gvisor; the RuntimeClass uses the pinned runsc runtime. Enabling it changes the default Karpenter node class from Bottlerocket to AL2023 so the runtime can be installed during node bootstrap. The AL2023 node configuration preserves single-process OOM killing for workload containers.

The default hawk:ciliumExclusive: "false" keeps Cilium chained to AWS VPC CNI and retains the default RFC 1918 Cilium pod pool. Exclusive mode makes Cilium the node CNI, uses an RFC 1918 overlay pod pool, and retires the aws-node DaemonSet after Cilium is ready. Treat that as a CNI migration for existing clusters: plan node recycling and a rollback path before enabling it.

Audit Logging¶

Application-Level Logging¶

Hawk API — all API requests are logged to CloudWatch with user identity, action, and resource context
Middleman — model API calls are logged with user identity, model (public name only), and token usage. Request/response bodies are not logged.
Token Broker — credential exchanges are logged with user identity and requested scope

AWS CloudTrail¶

CloudTrail is enabled by default in AWS accounts and logs all AWS API calls. CloudTrail Insights (anomaly detection for API call rates and error rates) can be enabled separately via the infra-shared repository.

VPC Flow Logs¶

VPC flow logs are enabled for all traffic and sent to CloudWatch Logs at /aws/vpc/flowlogs/<env>. Retention follows hawk:cloudwatchLogsRetentionDays (default: 14 days).

Endpoint Protection (CrowdStrike Falcon)¶

Hawk optionally deploys the CrowdStrike Falcon sensor to protect infrastructure hosts. Enable it with hawk:enableCrowdstrike: "true" and a Secrets Manager secret containing your CrowdStrike API credentials (see Configuration).

What's Protected¶

Target	OS / Arch	Installation Method
All EKS nodes (Karpenter)	Bottlerocket and AL2023	DaemonSet via falcon-sensor Helm chart (requires Falcon Images Download scope)
Tailscale subnet router	AL2023 / ARM64	Sensor RPM installed via cloud-init at boot

EKS nodes get the sensor from the DaemonSet and only from the DaemonSet. GPU nodes once also installed it as a host RPM from EC2NodeClass userData; they no longer do, because a node cannot run both — the containerized sensor exits immediately on a host that already owns /opt/CrowdStrike, leaving its pod in CrashLoopBackOff. Hosts outside EKS have no DaemonSet to rely on and still install the RPM.

The DaemonSet pulls a container image from registry.crowdstrike.com, which requires the Falcon Images Download: Read scope — part of the Falcon Cloud Security with Containers add-on. The subnet router downloads its RPM from the CrowdStrike API at boot using the Sensor Download: Read scope.

Tailscale ZTA Integration¶

With the Falcon sensor on the subnet router, you can enable CrowdStrike ZTA with Tailscale to gate tailnet access based on device posture. This requires a separate API client with Hosts: Read and Zero Trust: Read scopes, configured in the Tailscale admin console under Device Posture Integrations. Note that ZTA scores apply to the router instance itself, not to traffic routed through it.

AWS Security Services¶

AWS security services (GuardDuty, Security Hub, AWS Config, CloudTrail Insights) are managed by the infra-shared repository, not by Hawk. See infra-shared for configuration details.

Recommended Security Configuration¶

For production deployments, consider:

Setting hawk:eksPublicEndpoint: "false" and using Tailscale for private cluster access. If the public EKS API endpoint must remain enabled, set hawk:eksPublicAccessCidrs to restrict it to approved source CIDRs; omitting it preserves AWS's 0.0.0.0/0 default.
Setting hawk:albInternal: "true" to make the ALB private (requires VPN)
Setting hawk:protectResources: "true" to prevent accidental deletion of stateful resources (S3 buckets, secrets, the Datadog log-archive bucket, and the Aurora cluster)

Monitoring & Observability¶

CloudWatch¶

All services log to CloudWatch by default. Log retention is configurable via hawk:cloudwatchLogsRetentionDays (default: 14 days).

Key log groups:

Hawk API logs
Middleman logs
Lambda function logs
GuardDuty findings (when enabled): /aws/events/guardduty/<env>
Security Hub findings (when enabled): /aws/events/securityhub/<env>

Datadog (Optional)¶

For richer monitoring, Hawk supports Datadog integration:

hawk:enableDatadog: "true"

This enables:

APM — distributed tracing across API, Middleman, and Lambda functions
Log forwarding — CloudWatch logs forwarded to Datadog
Custom metrics — token usage, import counts, evaluation durations
Sensitive data filtering — danger_name, API keys, and auth headers are scrubbed from all telemetry

Budget Alerts¶

Monitor AWS spending with budget alerts:

hawk:budgetLimit: "10000"
hawk:budgetNotificationEmails:
  - "team@example.com"
hawk:budgetNotificationThresholds:
  - 80
  - 100

External Dependencies¶

Hawk interacts with the following external services:

Service	Purpose	Required?
Docker Hub	Pull base container images	Yes (login recommended to avoid rate limits)
LLM Providers	Model API calls via Middleman	At least one provider API key required
OIDC Provider	User authentication	Optional (Cognito used by default)
Datadog	Monitoring and observability	Optional
Slack	Budget alerts	Optional
CrowdStrike Falcon	Endpoint protection for EKS nodes and subnet router	Optional
Cloudflare	DNS delegation	Optional
GitHub	CI/CD via Pulumi Deploy	Optional

Network Security¶

TLS everywhere — all external traffic uses TLS via ACM certificates
Private subnets — EKS nodes, RDS, and ECS tasks run in private subnets with no direct internet access
NAT Gateways — outbound internet access from private subnets goes through NAT gateways
Security groups — restrict traffic between components (ALB → ECS, ECS → RDS, etc.)
VPC endpoints — S3 traffic stays within the VPC via a Gateway endpoint
Cilium network policies — pod-level network isolation within Kubernetes