Secure Token Service

Restricted credentials limit each eval or scan execution to approved model calls, input and output data, and runner infrastructure. A key mounted from a Kubernetes Secret authenticates JWT issuance and completion reports. Hawk stores its hash and an immutable grant. Middleman accepts the JWT for inference; Token Broker combines it with projected Kubernetes identity to issue scoped AWS sessions. Hawk's human-facing data APIs, including warehouse queries and presigning, reject restricted access tokens. Restricted executions access data through brokered S3.

Credential mode

The operator's restricted_credentials_by_default AppConfig flag selects the default for hosted evals and scans; its packaged default is false (user credentials). Override the default per submission with --restricted-credentials or --user-credentials. API clients use credential_mode="restricted" or "user"; omission or null uses the server default. Explicit choices do not read AppConfig; an unavailable configured store returns 503 for an unspecified choice. Packaged defaults do not overwrite an existing AppConfig value. Registered human evaluations retain their existing route.

With explicit restricted mode, the CLI and Python client omit the user's refresh token from submission and resume requests, including grouped scans. The access token still authenticates the submission. User or omitted mode still forwards the refresh token because the server may select user credentials.

A resume must use the job's existing credential mode. If the server default differs, pass the matching flag. A different mode returns 409; use a new job ID to change modes. Deleting a job does not reset its mode. A reservation that has never committed a grant or attempted a launch may still use either mode. Switching the default does not convert existing jobs: user-mode jobs need --user-credentials to resume after the default becomes restricted.

With user credentials, nonempty eval transcripts.sources and any non-null access_groups (including []) return 422. Empty eval sources and omitted/null access groups are allowed; ordinary scan sources remain supported.

Deploy compatible issuer, Middleman, token broker and runner images before accepting restricted submissions. Middleman must resolve accounting attribution even for model-free restricted executions. Restricted launches and resumes require a configured token_service_issuer, nonempty token_broker_url (HAWK_API_TOKEN_BROKER_URL), and trusted HTTPS runner_token_service_api_url (HAWK_API_RUNNER_TOKEN_SERVICE_API_URL). The API URL must be a bare origin with a hostname, no credentials, query or fragment, and no path except an optional /. If token_service_issuer is absent, restricted launches and resumes return 422, "Restricted credentials aren't enabled on this deployment." With an issuer configured, a missing broker URL or a missing or invalid runner API origin returns 503, "Restricted launch is not configured". Pulumi normally supplies these URLs; custom deployments must configure them. These requirements do not apply to submissions using user credentials or registered human evaluations. Custom images and execution virtualenvs must contain the updated Hawk package, its hawk_credentials entry point and hawk.runner.restricted_startup module. Updating an image does not update already-running workers.

Restricted Token Broker requests always require a valid projected Kubernetes token as well as the restricted JWT. Configure the broker's projected-token issuer, audience, and verification keys before enabling restricted launches. The requireJobToken / REQUIRE_JOB_TOKEN setting controls enforcement on the legacy broker path; it does not disable verification for restricted requests. Changing enforcement does not revoke already-issued AWS sessions.

See the credential-testing guide for local coverage and live validation.

Online scan: configuration is rejected in restricted mode; submit a standalone scan. Tasks using Hawk's human-facing APIs must use declared inputs and brokered S3 instead. Direct SDK token refresh and Scout's cross-pod resume-buffer persistence remain separate concerns.

New logical job IDs matching ex- plus exactly 32 lowercase hexadecimal digits are reserved, including IDs derived from names and explicit resumes. Other ex- names remain valid. Legacy Helm operations separately reject names that normalize to ex- plus 32 lowercase hexadecimal digits, including historical aliases. Restricted execution names are fresh on every launch; logs, status and metrics retain the logical job ID.

Provision database users before applying the normal migration chain.

Credentials and permitted access

Each restricted execution has one immutable grant with the SHA-256 hash of the complete mounted key. The key format is hawk_rc_v1.<grant UUID>.<secret>: a canonical lowercase UUID and 43 unpadded base64url characters encoding 32 random bytes, 91 ASCII bytes in total. The grant UUID is also the credential selector, OAuth client_id and JWT grant.grant_id. Plaintext keys never enter Helm values, saved configuration or the database.

Grant snapshots and signed payloads use schema version 1. The mounted key format is versioned independently. Never edit a saved grant to change its permissions or version. Permissions use sorted, deduplicated service:action@subset strings:

Atom	Capability
middleman:call@models/<public-name>	Invoke the exact approved public model name.
hawk:read@eval-sets/<id>	Read the whole named eval set, including later files.
hawk:read@scan-runs/<id>	Read this scan's own output.
hawk:write@eval-sets/<id>	Write this eval execution's own output.
hawk:write@scan-runs/<id>	Write this scan execution's own output.
hawk:use@runner	Use execution-bound infrastructure capabilities.
s3:read@<storage-name>	Read approved named external storage; evals only.

Unknown combinations, group selectors, wildcards and brace expansion are rejected. Resource names use urllib.parse.quote(name, safe="/"); decoding once and re-encoding must reproduce the spelling. Escaped characters are literal. Model names retain case; eval/scan IDs use existing validators. Storage names are 1–32 lowercase alphanumeric/hyphen characters from the deployment catalog, never literal buckets or paths. Callers cannot supply arbitrary atoms.

Models, attribution and output restrictions

At launch, Middleman authorizes each requested public model name against the submitter's current groups and returns its name and access group. Hawk captures those groups in the output restrictions and saves exact model-call atoms. Grants and JWTs contain no separate model-binding records or upstream target names. Model-free grants have no model-call atoms. Reading transcripts never approves model calls.

Public names are stable codenames and must not be reassigned to other models. Middleman allows a signed name while it exists in the registry and is not marked dead. It does not compare saved target, group or lab metadata. Declassification therefore does not interrupt an existing execution or weaken its saved output restrictions. Renewal copies the approved atoms; the 30-minute token lifetime does not recheck the submitter's access. An explicit resume gets fresh approval.

Submission resolves Inspect descriptors to public registry names while preserving the original runner configuration. none and none/none remain placeholders; other descriptors that resolve to a placeholder, such as openai/none, return 422. Use a different registry alias for a remote model.

Local hardcoded and mockllm providers need no remote invocation approval. Their named output classifications and inherited restrictions remain, and a configured remote default still needs approval. mockllm/... keeps its full identity. A remote descriptor such as openai/mockllm/model still needs remote approval. Local-only restricted executions still resolve submitter/team attribution; cross-lab validation uses the same shared policy for eval inputs and scans.

Every restricted submission and resume requires complete code inventory, source and named storage authorization, and Middleman attribution for the current submitting user. Resolution requires a user token and team attribution. Teams are the canonical list extracted under the authenticated provider's configuration. An empty list means no attributable teams under that configuration, including disabled extraction or a missing claim. Missing authentication returns 403, invalid or mismatched accounting identity 502, and resolver timeout 503, before launch writes.

Required usage_attribution holds the submitter's user_id and normalized teams, plus optional authenticated user_email; its user must match submitted_by. Missing/null attribution or teams is invalid. Renewal keeps this snapshot despite later ownership/team changes; a resume gets the new submitter's attribution while preserving output ownership. Accounting grants no permissions. Traffic user_id remains the signed execution subject; usage_user_id/usage_teams identify accounting and priority admission. The EMF display label uses an ASCII email when available, falling back to user ID, for both restricted and direct traffic. Authorization, accounting joins and priority flow identity continue to use IDs. The user's direct requests and executions share a priority flow within each priority class and quota group; restricted executions gain no user groups or admin rights.

transcripts.sources declares eval inputs for both evals and scans. All scan sources are candidates; after deduplication and excluding an eval's own output, at most 20 external sets are allowed. Evals reject source filters; scan filters and limits select work, while authorization covers whole folders/files, including CoT. The runtime helper exposes source identities and directories in declaration order, including duplicates, before factories load and on resume. Scout's selected transcript files are carried separately and remain its scanning worklist. HAWK_TRANSCRIPT_SOURCES is metadata, never authority, and does not widen that worklist. Inputs stay in S3.

Saved filtered-scan configs without separate source metadata must be regenerated through an explicit resume.

The restricted eval example combines transcript sources and declared access groups. Replace its source ID, model and groups with values available in your deployment before running it.

Eval and scan inputs follow the shared cross-lab policy and operator-controlled override. These checks run before launch writes or resume cleanup. They are separate from inherited output restrictions; source-read access never grants model invocation.

The API must be able to read every requested source's stored classification. A database role subject to row-level security can approve sources that are fully visible to it. Sources with hidden classifications are refused even if the submitting user has broader access. Membership in rls_bypass is not a blanket requirement for restricted launches.

Output restrictions include invoked-model groups and source model/code groups. Omitted access_groups inherits the required union; explicit [] asserts that none are required. Declared groups must cover the union, including previous output restrictions, and may add restrictions. The submitter must be authorized for the full set of output restrictions, including extras, before saving the grant. Resume saves these prior restrictions before replacing code inventory, so a refused declaration cannot weaken existing outputs. Users reading outputs need the inherited model and code permissions; later policy changes do not rewrite grants. The snapshot stores effective groups, without per-input/declaration copies.

An input-folder approval lasts for that consumer execution, including credential renewal. It covers files written later under the same source eval-set ID, even if a new source execution adds model or code restrictions. New and resumed consumers must satisfy the current source restrictions. Successful eval submissions that add restrictions under an existing ID warn about these earlier approvals; use a new eval-set ID when the new results must be isolated from already approved consumers. The warning compares the prior output restrictions with the final approved model and code groups, including inherited and explicitly added restrictions.

Every eval resume reminds users to declare models called by code. A missing declaration requires a config update and another resume. Declarations do not change a monitor's failure policy; task owners must configure it separately.

Relay attachment requires write access to every output model group and code-access-<group> for every output code group, including inherited restrictions. The API copies the approved output floor to the Job and runner Pod's inspect-ai.metr.org/restricted-access-groups annotation. Restricted Pods with an execution-ID label and a missing or malformed floor cannot be attached to. Ordinary unlabelled user Pods retain their existing model-access checks.

Brokered data and inference

Every grant includes reads of its own output; evals and scans can additionally read their declared eval-set inputs. Read atoms include object reads and scoped listing; own-scan reads include tag reads needed for finalization. Writes require a separate atom for the exact own output and job type; a read-only grant never gains writes. Writes permit output replacement/deletion and scan finalization tagging, with denies on API-owned metadata. The runner profile supplies configured KMS, ECR, telemetry and existing task-secret namespace capabilities, not data reads/listing or new secret isolation. It is required for broker eligibility together with projected identity.

The broker derives policies from signed data/storage atoms and uses every signed source eval set. Non-null request eval_set_ids lists, including [], are rejected for restricted executions; source selection with user credentials is unchanged. Evals can also use runner.storage_grants / runnerStorageGrants: approved names are intersected with the current deployment catalog. Restricted sessions do not reread .storage-grants.json; changing that file does not revoke their storage access. Removing a catalog entry excludes it from new sessions; resume obtains fresh approval. Sessions using user credentials retain the storage-gate and current-permission checks. publicModelsOnly checks still apply to restricted executions. A verified scan can ignore the dynamic __unresolved__ marker on its own output so it can finish writing. Source eval sets, other groups, and immutable output floors remain enforced. Eval/scan base policies cover infrastructure and approved reads/listing; separate write policies require an own-output write atom. Secret and named-storage policies retain their own gates.

Middleman supports these restricted routes:

Route	Restricted behavior
/anthropic/v1/messages, /anthropic/v1/messages/count_tokens	Exact authorized model.
/openai/v1/chat/completions, /openai/v1/completions	Exact authorized model.
/openai/v1/responses	Exact authorized model; foreground/streaming only.
/openai/v1/responses/input_tokens, /openai/v1/responses/compact	Exact authorized model.
/gemini/v1beta1/publishers/{publisher}/models/{model}:{operation}	Generate/stream/count for exact authorized model.
/google-ai/{api_version}/models/{model}:{operation}	Generate/stream/count for exact authorized model.
/openai/v1/models	Only live models named by signed model-call atoms.

File uploads, legacy unified inference and other catalog/query/control/admin endpoints require user credentials. Batch and background follow-up routes are not exposed. Responses require background absent or literal JSON false, checked before provider work. Restricted requests reject Anthropic fallbacks and OpenAI-compatible chat/Responses models (OpenRouter model fallbacks), including empty or null values, before credential lookup or upstream dispatch. Inline media remains supported; Inspect's Google Files API workflow does not. There is no conversion or fallback to user credentials. Clients must send the current JWT. Expiry is checked at admission; an admitted stream can finish. The mounted key cannot authenticate inference.

Restricted requests may declare tools the caller executes and a reviewed set of provider-run tools. Tools that call a model no grant approves (Responses image_generation, Anthropic's advisor, OpenRouter plugins) or read data stored with the provider by reference (file search and retrieval stores, stored prompts, cached content, containers reused by ID, files or skills attached to a container) fail with 400 before credential lookup, on token-counting routes too. So do nested model selectors (Vertex routingConfig and modelConfig), Gemini extras on Vertex's OpenAI-compatible endpoint (extra_body or google, which can carry cached content and Gemini tools), remote MCP servers and types Middleman does not recognize, which stay unavailable until reviewed. Provider-run web search, web fetch, URL context and code execution in a new container are allowed: runners can reach the internet themselves, so these add no new outbound path. The per-provider allowlists live in restricted_tools.py.

Anthropic container IDs are accepted, because Inspect resumes its own container when web search with dynamic filtering or code execution pauses a turn. Like other provider object references in request content (OpenAI previous_response_id, conversation and file_id parts, Anthropic file sources, Gemini fileData URIs including Vertex gs:// paths), they are not yet bound to the execution that created them.

Configure and rotate

Set hawk:tokenServiceEnabled to true to provision signing resources and configure token issuance. It defaults to false, including in development environments; development stacks do not inherit enablement from staging. Provisioning does not select restricted credentials for new launches. Without issuer configuration, token issuance and JWKS return 503 temporarily_unavailable. Completion and cleanup remain available and require the database.

Pulumi manages RSA-2048 SIGN_VERIFY KMS keys and aliases, with a 30-day deletion window. Only the API runtime role gets kms:Sign, scoped to the active ARN and RSASSA_PKCS1_V1_5_SHA_256. Provisioning exports public keys; requests need no kms:GetPublicKey, and Python never receives a private key.

Every generation's key policy also restricts signing to the API tasks role and that algorithm, including signing authorized through KMS grants. The API's IAM policy selects the active key. Account administrators retain policy-management and deployment permissions, including public-key reads, alias management and scheduled deletion. This protects against broad signing permissions; a trusted administrator can deliberately change the key policy. Key-policy changes require checking issuance, deployment administration and rotation before rollout.

Keys and aliases follow hawk:protectResources. Non-development stacks default to protected; development stacks are unprotected. The teardown procedure disables protection before destruction and removes the stack only after destruction succeeds.

Pulumi setting	Default	Constraint
hawk:tokenServiceEnabled	false	Provision signing resources and configure issuance. Development stacks must opt in locally.
hawk:tokenServiceSigningKeyGenerations	[v1]	Nonempty list of unique, nonempty generation names.
hawk:tokenServiceActiveSigningKeyGeneration	v1	Must be published.
hawk:tokenServicePublishedSigningKeyGenerations	All managed generations	Optional nonempty, unique subset of managed generations.

All API replicas share HAWK_API_TOKEN_SERVICE_ISSUER with issuer, audience, jwks_uri, kms_key_arn, kid and jwks. For canonical <api-url>:

Field	Value
issuer	<api-url>/token-service
audience	<api-url>/token-service/services
jwks_uri	<api-url>/.well-known/jwks.json
kid	Immutable KMS key UUID, never an alias.

Middleman's MIDDLEMAN_TOKEN_SERVICE_AUTH and broker's TOKEN_SERVICE_VALIDATION carry the same flat issuer/audience/jwks_uri object. There is one restricted audience; user OIDC and projected Kubernetes audiences are independent. Missing Middleman configuration permits only user credentials; malformed settings or an issuer also configured for user credentials fail startup.

Signing-key rotation and JWKS caches

Follow the deployment procedure, including logging in to the S3 Pulumi backend before Pulumi commands. Rotate in separate deployments:

1. Publish a never-used generation such as v2 alongside v1, keeping v1 active. Verify every API replica publishes both keys.
2. Wait at least five minutes after replicas agree before switching signers.
3. Activate v2; verify old signers have stopped and old-key signing permission is removed. Keep the old public key published for at least one hour after the last old signer stops. IAM changes can precede rolling task replacement, causing transient 503s; clients retain valid JWTs and back off. An ECS rollback alone does not restore old signing permission: finish the rollout or revert the active generation and redeploy.
4. Remove old public-key publication while retaining its managed key. To schedule deletion later, explicitly unprotect both key and alias if protection is enabled, then remove the managed generation; the key enters its 30-day deletion window.

Never reuse a generation or kid. Emergency removal requires stopping old signers and removing their public key from every replica; caches and issued capabilities mean this is not instant revocation.

Both consumers fetch only their configured JWKS URI, fetching keys on demand and caching parsed RSA keys locally per process/worker. They do not trust general shared Valkey, whose other writers could otherwise forge restricted identities. A successful fetch is fresh for 300 seconds; known keys can survive an outage until an absolute 3,600-second hard deadline. Cache hits extend neither deadline; a successful replacement removes keys missing from the new document. Unknown kid refreshes are throttled to once per 30 seconds per instance. A successful fetch proves absence, including during the cooldown while that document remains fresh. A failed refresh reports issuer unavailability for unknown keys even if an older document remains fresh; usable known keys retain the outage fallback. Fetches are limited to ten seconds and 64 KiB. Middleman returns provider-shaped 503s with Retry-After: 5 for issuer outages, not invalid-key errors.

Broker warm invocations reuse one event loop and verifier runtime. Replacing that runtime starts a fresh JWKS cache; cache entries and refresh throttling are not transferred between clients or loops.

Hawk formats the fixed RS256 restricted access token header and validated claims, then awaits KMS signing directly. KMS signs the SHA-256 digest of that exact input; Python never receives a private key. The signer checks the returned key ARN, signing algorithm and RSA-2048 signature length. Prelaunch and issuance size checks use the same serialization and fixed signature length. joserfc imports public keys and verifies JWT signatures and registered claims.

JWTs use RS256, typ: at+jwt, strict claims and configured issuer/audience, a maximum 8,192 encoded bytes, and 60 seconds of clock skew. The broker validates both JWT and projected token locally without reading execution lifecycle. It matches saved namespace/ServiceAccount names and checks projected signature, issuer, audience, expiry and subject. ServiceAccount UID and Pod name/UID claims must be nonempty strings; these are token-shape checks, not stored Hawk UID bindings.

Decommissioning

Disabling hawk:tokenServiceEnabled removes the signing resources, JWKS publication, and consumer validation configuration in the same deployment. There is no separate drain switch: changing the default for new launches does not prevent explicit --restricted-credentials launches. Coordinate a maintenance window with submitters, and leave the setting enabled until the final step.

1. Confirm no restricted execution is live. Run this query as a database administrator with access to all execution and grant rows. It must return no rows; otherwise, let those executions finish or end them through Hawk before proceeding.

SELECT e.pk AS execution_id, e.job_pk, e.namespace_name, e.kubernetes_job_name
FROM public.job_execution AS e
JOIN token_service.execution_grant AS g ON g.execution_pk = e.pk
WHERE e.lifecycle_state = 'active';

1. Wait the maximum token lifetime plus one hour. With the current 30-minute JWT lifetime, wait at least 90 minutes after confirming that no restricted execution is live. Keep issuance and verification enabled throughout the wait. If another restricted execution starts or resumes, repeat step 1 after it ends and restart the wait. Recheck before proceeding.
2. Unprotect only the signing keys and aliases. Follow the deployment procedure to log in to the Pulumi backend. List the resources, then copy the key and alias URNs for every managed token-service signing generation, including retired generations. Replace the placeholders below with the target stack and those URNs:

pulumi stack --stack '<stack>' --show-urns
pulumi state unprotect --stack '<stack>' '<signing-key-urn>'
pulumi state unprotect --stack '<stack>' '<signing-alias-urn>'

Repeat the unprotect commands for each generation. Do not set hawk:protectResources=false or use --all: that would also unprotect unrelated resources such as RDS and S3. Do not deploy between this step and disabling the token service, because a deployment can restore protection. 4. Deploy with the token service disabled.

pulumi config set hawk:tokenServiceEnabled false --stack '<stack>'
pulumi up --stack '<stack>'

Scheduling KMS key deletion stops signing immediately. The 30-day deletion window is a recovery period, not an additional serving period, and re-enabling is not guaranteed to restore the same key identity.

The setting does not delete database authority records. Issued JWTs and acquired AWS credentials retain their separate expiry bounds after issuance stops.

Operate and recover

Launch, bootstrap and renewal

Under the logical launch lock, the API commits a restricted execution and grant, publishes output restrictions, installs Helm, then creates the required immutable key Secret in the execution namespace. Active means authority is open, not that its Pod started. Runner failures require explicit resume.

Failed or cancelled launches close authority and request cleanup. Cancellation cannot advance from Helm to key publication. Uncertain cleanup blocks replacement; confirmed cleanup preserves the logical job for resume. See Delete and resume.

The launcher protects these settings from user overrides and preserves them across installation and exec:

Variable	Value
HAWK_RUNNER_RESTRICTED_API_URL	Trusted HTTPS origin, no user info/query/fragment or path except /.
HAWK_RUNNER_RESTRICTED_EXECUTION_ID	Canonical execution UUID.
HAWK_RUNNER_RESTRICTED_KEY_FILE	Absolute path to the read-only mounted key.
HAWK_RUNNER_RESTRICTED_ISSUER	Configured issuer, independently of API URL.

User-supplied environment variables and secret declarations may not use the HAWK_RUNNER_RESTRICTED_ prefix. The API returns 422 before launch in either credential mode. This includes request secrets, runner.environment, and the target variable names of AWS-backed secrets. A saved scan config containing such a name must be corrected and submitted under a new ID; scan resume cannot replace that config.

If all four are absent, the runner uses user credentials. Partial or invalid restricted settings fail startup. The launcher also provides:

Variable	Value
HAWK_RUNNER_RESTRICTED_MIDDLEMAN_URL	Trusted Middleman base URL, required by bootstrap.
HAWK_RUNNER_RESTRICTED_MANAGED_KEYS	JSON [] initially; bootstrap records generated API-key environment variable names. Preserve across exec.
HAWK_RUNNER_RESTRICTED_CACHE_DIR	Chart default /tmp/hawk-restricted; otherwise a hawk-restricted-<uid> directory under the system temporary directory.

Before task/scanner construction, HAWK_RUNNER_REQUEST_CONTEXT carries job type, job ID and priority to workers for request headers. This metadata contains no credentials and grants no access, and is available with telemetry disabled.

Bootstrap acquires credentials before secrets, dependencies or work. Bootstrap and post-install reacquisition each retry transient failures for up to 15 minutes. Recognized permanent OAuth rejections fail without falling back to user tokens; unexpected intermediary 4xx responses are transient. Persistent 404/405 responses suggest a wrong API origin.

Hawk does not automatically forward the submitter's human access or refresh token to restricted runners. Explicit provider credentials and base-URL settings retain precedence over generated provider settings. Those explicit credentials retain their own permissions and renewal requirements; the restricted grant governs Hawk-issued credentials.

Startup, Inspect hooks and the AWS helper share private credential storage. Restricted renewal is on demand when a token has at most ten minutes remaining; transient failures retain valid tokens with backoff. Inspect hooks return without waiting for network or peer locks; asynchronous authentication retries pick up credentials published by the background refresh. Startup and AWS helpers wait for acquisition. Scheduling budgets and socket timeouts are soft limits.

Provide writable private cache storage. Paths use numeric UIDs on POSIX, so a passwd entry is unnecessary. A storage failure may reuse a previously published, still-valid restricted token, except after authentication rejection or permanent OAuth refusal. Without one, startup fails immediately with a safe path diagnostic. The standard chart requires writable /tmp/hawk-restricted. Custom launchers can set HAWK_RUNNER_RESTRICTED_CACHE_DIR in the runner environment to another writable absolute directory; it is not a user config override. User refresh-token rotation requires successful publication. Process exit, network failure or failed publication after rotation may require fresh user credentials. A stuck living lock holder must finish or be restarted; process death releases the lock. The separate AWS session cache can return acquired AWS credentials even if its cache write fails.

Scout workers and fresh-process resumes discover the installed Inspect hooks. Changing mode, issuer, execution, key path or provider configuration requires a new process. Managed keys are replaced only for the configured issuer/execution; services still verify the JWT. Parent preparation supplies initial provider keys. Direct SDK clients that copy a token need their own renewal mechanism and may fail construction without an initial key.

Job checks and credential lifetime

Before each issuance, the API authenticates key/client ID, validates the saved grant and requires active state, then closes that database session before an exact Job GET through the monitoring provider's long-lived client. Final Complete=True / Failed=True or a Kubernetes 404 Status response with reason NotFound and details identifying batch/jobs and the saved Job name conditionally closes that execution while active, commits, and returns 401 invalid_client; it preserves an earlier terminal result and cannot close a replacement attempt.

A live Job leads to an execution-row lock and active-state recheck before claim capture. Closure during the GET prevents signing. Queueing, suspension, deletion timestamps and early FailureTarget / SuccessCriteriaMet conditions are not terminal proof; removing a ServiceAccount alone does not close authority. Unrecognized 404 responses, malformed/conflicting observations, read failures and timeouts return 503 without signing or permanent closure. Each GET has five seconds within the ten-second capture budget. Kubernetes remains a renewal dependency when launches are disabled. Timeout cancels and drains the read; the shared monitoring client remains open for other requests until API shutdown.

Claims and expiry are captured once; the transaction/session close before KMS signing. Signing consumes the 1,800-second JWT lifetime; expires_in reports what remains, and expired signing results are discarded. An issuance already authorized can finish after closure. Valid JWTs can still obtain AWS sessions: with the default one-hour session, nominal residual access is 90 minutes after recorded closure, plus skew and in-flight work. Reporting delays postpone closure itself. This is neither immediate revocation nor a universal AWS-session cap. Capabilities expire on their own schedules.

There is no periodic authority reconciler or launch-expiry timer. Without another token request or delivered completion, authority can stay active indefinitely; explicit deletion recovers it. The hourly janitor performs resource housekeeping, using inspect-ai.metr.org/helm-release for physical releases and job-id as the legacy fallback. Job checks never adopt, repair or delete resources.

Completion and retained runners

The hosted process wrapper reports after work, final writes and teardown finish or unwind; scan reporting follows closure of the work event loop. Startup failures use the same exit guard; successful exec transfers ownership to the replacement. Direct Python work-function calls do not report process completion.

Restricted runners stop accepting new CloudWatch metric records at shutdown and wait at most five seconds for their daemon. Unsent buffered metrics are discarded (normally up to the 60-second flush interval); an already-running flush or credential operation can finish after completion. Evals using user credentials flush buffered metrics at eval-set completion. Eval results and Middleman token-usage accounting are unaffected.

Completion rereads the mounted key each attempt, sends exact {}, and needs no JWT/STS session. Redirects and environment HTTP proxies are disabled. A valid acknowledgement names the same execution, ended state, a nonempty terminal reason and an aware ended_at; additional response fields and new reasons are accepted. The first terminal result wins. runner_completed means finished authority use, not successful work or stopped resources. Delivery failure does not rerun work or change its result. KMS/issuer configuration is unnecessary for completion, but the database must be available.

The synchronous delivery loop starts immediately, retries after 1/2/4/up-to-five seconds, and stops scheduling work after 30 seconds or a validated acknowledgement. HTTP attempts use five-second socket-phase timeouts, read at most 16 KiB and check the remaining budget between response chunks. An operation already underway can exceed the budget. Ctrl-C stops reporting immediately and can replace an earlier work failure with exit 130. Failed client cleanup can cause a harmless idempotent retry. A validated acknowledgement stops further reports.

Restricted evals with cleanup: false report before retained debugging. A stop during reporting prevents idle entry. Until acknowledged, idle retries once every 60 seconds even after JWT/STS expiry, without S3 stop polling or consumer renewal. No new consumer renewal starts after final writes; an already-running refresh may finish. Leaving idle opens no new 30-second window. Retention has no automatic deadline: use hawk delete <job-id> when done. Scans gain no retention; retained runners using user credentials keep S3 stop behavior. If all reports fail, a retained live runner may stay authorized until delivery or deletion.

Delete and resume

A DELETE that overlaps a launch or another deletion returns 409. No deletion is queued: retry the same hawk delete command after the other operation finishes. The client does not automatically retry this conflict. Retryable cleanup failures remain separate and use the existing 503 handling.

Select the matching credential mode when resuming. The following tracked cleanup checks apply to restricted executions.

Resume checks the previous Job even when its execution is still active because the runner crashed without reporting completion. A terminal or missing Job with physically stopped Pods can be cleaned during resume. Cleanup commits authority closure and deletion intent before removing resources; an active execution ends with deletion_requested, and an existing terminal reason is preserved. A Kubernetes observation failure returns a reportable, retryable 503 without starting cleanup. Live retained runners or unrecoverable execution history (such as a failed launch with no deletion intent) require hawk delete <job-id> before cleanup can begin. Once deletion intent is committed, another resume can continue cleanup, including after the previous cleanup timed out. If cleanup remains pending, resume returns 503 with Retry-After: 5; retry resume or use hawk delete. The CLI does not automatically retry resume submissions. Every restricted resume gets a new execution, key and release only after confirmed cleanup. A scan whose accepted config was never published still cannot be resumed.

Under the launch lock, known output, model and input restrictions are checked against declared access_groups before resume cleanup. Missing groups are named only after the submitter is authorized for every required group. Final approval checks current code policy and database floors again after inventory certification. The config and its associated metadata are published only after final approval, grant-size validation and effective-floor publication, before Helm. Rejected approvals preserve the saved config. Physical resume cleanup records resource absence without marking the logical job deleted, including after a failed replacement launch; explicit deletion still does. Later failures can leave physical cleanup or conservative access gates in place.

Delete commits closure before external cleanup, validates saved layout/namespace ownership, uninstalls the saved release and removes surviving expected namespaces. Namespace deletion removes the Secret; cleanup does not separately read/delete it, the Job or ServiceAccount. Any surviving resource subset is recoverable, including authority committed before Helm created anything.

A successful DELETE means authorization closure is committed, the saved release was uninstalled or absent, and namespace deletion requests succeeded. Namespaces may still be terminating. The logical job is marked deleted and ancillary cleanup is attempted; success does not assert that all resources are gone.

Confirmed absence requires every expected namespace to be gone and an exhaustive execution-label listing to show none remaining. That separate record permits a restricted replacement launch. Wrong ownership or read/delete/list failures remain reportable 503s. Repeating an accepted DELETE preserves its timestamps; confirmed cleanup requires no further Helm/Kubernetes calls.

Resume waits for confirmed absence and returns 503 with Retry-After: 5 if cleanup remains pending. Retry resume as described above. Normal public DELETE does not return this pending-cleanup error.

Expected termination-in-progress responses log at WARNING without a Sentry error event, including inline resume cleanup. Waiting stops on a successful observation of surviving namespaces after 55 seconds, leaving a five-second request allowance inside the overall budget. Dependency failures, ownership mismatches, and ambiguous timeouts remain reportable and retain the retry header. Resume conflicts retain their 409 status; they are not converted into cleanup retries.

The CLI and Python client's delete_job automatically retry 503 responses, honoring numeric or HTTP-date Retry-After within the client's timeout (300 seconds by default). HTTP dates without a timezone are interpreted as UTC. Persistent failures still surface for repair and retry. Never delete the shared Helm namespace. Execution namespaces, including sandboxes, carry inspect-ai.metr.org/execution-id; listing proves absence, not permission to delete/adopt arbitrary namespaces. Eval monitors are removed best effort; preserve legacy human-eval assignments, jump-host cleanup and job-level monitoring.

Helm uninstall is not UID-conditional. Safety relies on never-reused execution names, launch/delete locking, deletion barriers and control-plane RBAC. Runners cannot mutate their runner namespace, Job or ServiceAccount. Trusted administrators recreating identical names are not detected by stored UID comparison; direct namespace deletes use the UID observed during that attempt.

Completion/delete preserve the logical Job, execution/grant history, saved names and outputs. They cannot narrow restricted_access_groups inside .models.json; there is no separate access-groups file.

Endpoint and failure reference

Use TLS and authorization headers, never credentials in URLs. Both credential endpoints limit bodies to 4,096 bytes and return Cache-Control: no-store and Pragma: no-cache, including errors.

Endpoint	Authentication and request	Success
POST /oauth/token	HTTP Basic: grant UUID username, complete mounted key password. application/x-www-form-urlencoded body grant_type=client_credentials.	200: access_token, token_type: Bearer, expires_in; no refresh token.
POST /token-service/executions/{execution_id}/complete	Bearer mounted key, not JWT; application/json body {}; no query parameters.	After commit, 200: execution_id, lifecycle_state: ended, end_reason, ended_at; repeats return the original terminal result.
GET /.well-known/jwks.json	Public.	Configured public JWKS; Cache-Control: public, max-age=300, no DB/KMS request.

OAuth Basic form-encodes each component before joining with : and base64 encoding; ordinary OAuth clients support it. Posted credentials or combined auth methods are rejected. Empty/missing grant type is invalid; empty scope is accepted, any nonempty scope (including whitespace) rejected. Unknown form and harmless query parameters are ignored; credential-bearing queries and duplicate form fields are rejected before flattening. Limits: 64 form fields and a 1,024-byte token Authorization header. The saved grant supplies permissions, not negotiated scopes.

Forwarded HTTPS is trusted only through accepted proxy middleware. The deployment trusts forwarded peers broadly and relies on restricted ingress to limit them to the proxy; preserve that boundary. Plain HTTP token requests fail even on loopback. Each Uvicorn process allows eight issuance requests, including directly awaited KMS signing. Capture has ten seconds; cancellation waits for rollback/session cleanup and can exceed that budget. Normal Uvicorn shutdown drains requests before closing dependencies; forced shutdown is best effort. Limits are initial defaults, not measured production capacity.

Errors have JSON error; 401 includes Basic/Bearer challenge as appropriate with realm hawk-token-service. Every credential 503 below has Retry-After: 5.

Condition	Status	error
Missing/malformed/wrong key or identity; issuance for ended execution	401	invalid_client
Invalid/duplicate/missing fields, forbidden query, malformed/nonempty completion JSON	400	invalid_request
Plain HTTP token request	400	insecure_transport
Other grant type	400	unsupported_grant_type
Nonempty scope	400	invalid_scope
Wrong content type	415	invalid_request
Body over 4,096 bytes	413	invalid_request
Missing config, DB/KMS/Kubernetes outage, uncertain Job evidence, signing expiry	503	temporarily_unavailable
Corrupt saved grant or captured claims	500	server_error

Absent JWKS configuration also returns 503. Fix request errors before retrying; for transient renewal failures keep a valid token and back off. An unavailable issuer does not establish that an unknown signing key is invalid.

The broker uses separate error responses. For restricted AWS and GitHub broker requests, invalid or expired JWTs return 401; identity-binding or permission denials return 403; unavailable issuers, JWKS, or authentication configuration return 503. A restricted token on the human-only /validate route still returns 403.

Database authority and diagnosis

The hawk_api NOLOGIN role has SELECT/INSERT, no UPDATE/DELETE, on the private grant table. Hashes/snapshots are immutable. Runtime membership goes only to the actual API database login selected from the normalized first writer, without RLS bypass; public-only parent restrictions remain. Importers can write public and bypass RLS, but cannot read or forge grants in the separate token_service schema.

With publicModelsOnly, warehouse-backed API reads currently hide outputs with inherited private-code restrictions, even from callers with matching code grants. Code-only access in this mode is deferred. This limitation is specific to warehouse-backed reads; it does not change S3/folder authorization.

On PostgreSQL 17, the provisioner/migration administrator may retain membership with ADMIN=true, INHERIT=false, SET=false. Deployment audits all grantors; unexpected memberships or API delegation fail deployment and require explicit administrator repair. Remove old API-login membership before changing login. Inspect the grants before repair:

SELECT member.rolname AS member, grantor.rolname AS grantor,
       membership.admin_option, membership.inherit_option, membership.set_option
FROM pg_auth_members membership
JOIN pg_roles member ON member.oid = membership.member
JOIN pg_roles grantor ON grantor.oid = membership.grantor
WHERE membership.roleid = 'hawk_api'::regrole
ORDER BY member.rolname, grantor.rolname;

Execution, private-authority and output-access migrations extend the normal migration history as one chain.

Logger hawk.api.token_service emits bounded token_service_request events. CloudWatch EMF namespace Hawk/TokenService uses request_count/duration_ms with operation and result dimensions. Operations are issuance, completion, jwks; results are the error categories above plus success, not_found, method_not_allowed. OpenTelemetry spans are token_service.<operation>. Responses with 5xx status emit at ERROR with an allowlisted failure_stage: configuration, database, kubernetes_job, signing, or unknown. These events include the observed exception type and authenticated execution ID when available. Every event includes the generated request ID returned in X-Request-ID; IDs and exception types are not metric dimensions. Other responses remain INFO. Kubernetes dependency failures retain their original exception type and, when available, numeric dependency_status_code, so an RBAC 403 can be distinguished from a service 503 or timeout. Exception messages and response bodies are omitted. Cleanup failures add the same bounded diagnostic fields as warning breadcrumbs while reporting a sanitized error. Failures continue to produce Sentry events; this path does not suppress repeated failures during an outage.

Restricted AWS role sessions use hawk-<execution_id> from the verified signed claims. CloudTrail attribution follows the role session to the execution, its grant and submitted_by. Renewals of the same execution reuse the session name; individual credential issuance events are distinguished by AWS event and access-key identifiers. This attribution covers broker-issued sessions and does not promise propagation through subsequent AssumeRole calls.

Sentry retains credential events and transactions, but removes request bodies and query strings from /oauth and /token-service/executions requests. Its built-in scrubber removes authorization and named secret fields recursively. Generic key fields are scrubbed on credential routes; ordinary storage-key diagnostics are retained. Explicit restricted_key and credential_key fields are scrubbed on every route. Frame-local capture remains disabled; ordinary endpoints retain body/query diagnostics. Dependency logs are not suppressed. Restricted diagnostics never log exception messages, mounted keys, hashes, JWTs or Secret data. Do not put credentials in URLs or log messages: field scrubbing does not sanitize arbitrary text, and ALB and other edge logs are unaffected.