LLM Proxy (Middleman)¶

Middleman is Hawk's built-in LLM proxy. It runs on ECS Fargate and routes model API calls to providers (OpenAI, Anthropic, Google Vertex, DeepSeek, Fireworks, and more) with automatic token refresh and access control.

How It Works¶

When evaluations run on the cluster, Inspect AI sends model API calls through Middleman instead of directly to providers. Middleman:

Authenticates the request using the runner's scoped credentials
Routes the request to the correct provider API
Handles token refresh and retries
Enforces model group permissions

Callers use each provider's own API format through the passthrough routes (/anthropic/v1/messages, /openai/v1/chat/completions, /openai/v1/responses, /openai/v1/decisions, /gemini/...). The older unified /completions, /count_prompt_tokens and /embeddings APIs have been removed; /embeddings has no passthrough replacement. Models on labs that only the unified API served (gemini, gemini-public, vertex, dummy and dummy-chat) still load and appear in model listings, but no route can call them.

Setting Up API Keys¶

Provider API keys are set through Middleman's admin API (Granting admin below), wrapped by hawk proxy secrets set. The helper script takes KEY=VALUE pairs; the target deployment is selected by HAWK_MIDDLEMAN_URL, so run hawk login first:

scripts/dev/set-api-keys.sh OPENAI_API_KEY=sk-...

Set multiple keys at once:

scripts/dev/set-api-keys.sh OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-...

Setting a key rediscovers models straight away. The response reports the resulting catalog size and a per-provider outcome — ok, cached (upstream failed, serving the last good list), failed, no_key, not_run (cleared, and the reload has not refetched it yet), or stale (the reload failed before refetching it) — so you can tell whether the key took effect without reading the proxy's logs. hawk proxy models discovery shows the same without triggering a fetch; hawk proxy models reload forces one. A slow provider can outlast the response budget; the write still stands and the rediscovery keeps running, so re-check it with hawk proxy models discovery. If the reload fails, the write stands too; retry with hawk proxy models reload.

Supported Providers¶

OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, GOOGLE_APPLICATION_CREDENTIALS_JSON, DEEPINFRA_TOKEN, DEEPSEEK_API_KEY, FIREWORKS_API_KEY, HYPERBOLIC_API_KEY, META_API_KEY, MISTRAL_API_KEY, OPENROUTER_API_KEY, TOGETHER_API_KEY, XAI_API_KEY, plus OPENAI_HIGH_PRIORITY_API_KEY and ANTHROPIC_HIGH_PRIORITY_API_KEY.

High-priority requests use the dedicated high-priority key when present, otherwise the matching ordinary key. This also applies within Anthropic predeployment and OpenAI data-retention credentials; fallback never crosses those credential families. Explicitly configured credential-profile errors still fail.

Vertex-routed models require a GCP project

Models whose lab's upstream URL contains the {gcp_project} placeholder (the Vertex AI labs: gemini-vertex-chat, gemini-vertex-chat-global and vertex-serverless as seeded) need a GCP project: set a Google default profile (Google credentials), hawk:middlemanGcpProjectForPublicModels, or include a project_id in GOOGLE_APPLICATION_CREDENTIALS_JSON. Deployments with no such models don't need this. If one is configured without a project, Middleman fails to start. A model added later, by a reload, logs models.gcp_project_missing instead, and its requests fail with a 500 saying GCP project not configured.

Google credentials¶

By default Middleman calls Google (Vertex AI and the Gemini Developer API) with a token minted from GOOGLE_APPLICATION_CREDENTIALS_JSON, or with GEMINI_API_KEY on the Developer API route. Setting a Google default credential profile replaces both with a key-free token, like the other providers' aws_sts profiles: Middleman asks AWS STS GetWebIdentityToken for a JWT naming its task role, exchanges it at Google STS for the profile's workload identity provider, and uses the result directly (no service account). Every Google request then carries x-goog-user-project: <quota_project_id>, and {gcp_project} resolves to the same project. The token is requested with both the cloud-platform and generative-language scopes, so one profile serves Vertex and the Gemini Developer API.

hawk:middlemanStsWebIdentityEnabled: "true"
hawk:middlemanGoogleProfiles:
  models:
    workload_identity_audience: //iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>
    quota_project_id: <gcp-project-id>
hawk:middlemanGoogleDefaultProfile: models

The Google side needs, in that project:

a workload identity pool with an OIDC provider whose issuer is the AWS account's outbound web identity federation issuer (aws iam get-outbound-web-identity-federation-info), mapping google.subject to assertion.sub and with an attribute condition admitting only the middleman task role (arn:aws:iam::<account>:role/<env>-middleman-tasks; dev stacks use dev-<name>-middleman-tasks). Leave its allowed audiences empty: the default admits the provider's own resource name;
roles/aiplatform.user and roles/serviceusage.serviceUsageConsumer granted to the pool's identities;
the Vertex AI and Generative Language APIs enabled, and any Model Garden model a lab serves (such as a Llama MaaS model) enabled.

Use a separate project and pool per environment, each trusting only its own AWS account. Dev stacks inherit stg's profiles and default profile unless they set their own (Dev stacks and credential profiles), so stg's pool must also admit the dev task roles.

Labs and the key-host allowlist¶

A model's lab names a row in the middleman.lab database table. The row says which passthrough API serves the lab's models, the upstream URL to forward to, which provider key to send, the provider a model on the lab defaults to, and whether its models are chat models. Middleman loads the active rows together with the model configs, so a lab edit takes effect when the models reload. Editing the table directly in the database doesn't trigger one by itself (writes through /admin/labs/ do; see Managing labs), so run hawk proxy models reload after one (any model write also reloads). With the shared Valkey cache configured, the reload is broadcast and every worker reloads within moments; without it, only the worker serving the request reloads and the rest pick the change up at their hourly refresh, so restart Middleman to apply it everywhere immediately. The migration that created the table seeded one row per lab Middleman served at the time; models on labs no route served (dummy, dummy-chat, gemini, gemini-public, vertex) still load but stay uncallable, as before. Having no row, they now report no provider (previously google for gemini, gemini-public and vertex) and are listed as chat models (previously not, for dummy and vertex). Deactivate those models or move them to a served lab.

Because a lab's URL and key name are data an admin can change, Middleman only sends a provider credential to hosts allowed for it in deploy config. Built-in defaults cover every provider Middleman ships with (OPENAI_* keys and OpenAI credential profiles to api.openai.com, OPENROUTER_* (the key and OpenRouter credential profiles) to openrouter.ai, DEEPSEEK_API_KEY to api.deepseek.com, the GCP token to aiplatform.googleapis.com and *-aiplatform.googleapis.com, and so on; see middleman/src/middleman/key_hosts.py). To send a key to another host, set hawk:middlemanProviderKeyHosts to a JSON object of credential-name globs to host lists, e.g. '{"ACME_API_KEY": ["api.acme.example"]}'. An entry with a new glob adds hosts, but an entry spelled exactly like a built-in glob (such as OPENAI_*) replaces that built-in's hosts, so to add a host to a built-in entry, repeat its existing hosts: '{"OPENAI_*": ["api.openai.com", "openai.example.com"]}'. (A differently spelled glob, such as OPENAI_API_KEY, adds hosts for the names it matches.) Hosts match exactly or with a leading *. or *- wildcard for one DNS label, and URLs must be https:// with a DNS host name, so an IP address in the allowlist never matches a lab written through the admin API. A request whose key isn't allowed for its lab's host fails with a 500, and Middleman logs passthrough.key_host_not_allowed with the lab, the credential name and the host (never the credential). The Anthropic and Gemini Developer API routes use fixed hosts and aren't affected. Labs are managed with hawk proxy labs, which calls Middleman's /admin/labs/ API (Managing labs); the API checks the same allowlist when a lab is written, so direct API calls get the same check.

High Availability¶

For opt-in priority allocation, quota auditing and shadow validation, see Priority and per-user allocation.

By default Middleman runs a single replica. Setting hawk:highAvailability: "true" runs multiple replicas across AZs, which requires the shared Valkey cache (hawk:valkeyEnabled: "true" or an external hawk:valkeyUrl) — replicas share their provider-key/model caches and serialize secret writes through Valkey; without it they serve divergent caches and race secret writes. Deploys fail fast if HA is enabled without Valkey. See the configuration reference.

Bypassing the Proxy¶

To use your own API keys instead of Middleman, pass the matching key as a secret, disable the proxy's token refresh, and override that provider's base URL so it points directly upstream. For OpenAI, for example:

runner:
  environment:
    HAWK_RUNNER_REFRESH_URL: ""
    OPENAI_BASE_URL: https://api.openai.com/v1

Then pass your API key as a secret:

hawk eval-set config.yaml --secret OPENAI_API_KEY

Managing Models¶

Model configurations are stored in the database and organized into model groups for access control — a user must belong to a model's group to use it. A fresh deploy starts with an empty model registry, so hawk eval-set fails with Middleman error: Models not found until you add at least one model.

Models are managed through Middleman's admin API, wrapped by the hawk proxy models CLI commands.

Granting admin¶

The admin API is gated by is_admin=true. There are two ways to grant it:

Auth0 / Okta: emit the https://middleman.metr.org/claims/admin: true JWT claim. Manage admins in your identity provider.
Cognito (default open-source deploy): set hawk:middlemanAdminGroups in your Pulumi config to a non-empty list (e.g. ["middleman-admin"]) and add users to that Cognito group. The default is empty, so the group-based admin path is opt-in.

For Cognito:

# 1. One-time per stack: opt in to the group-based admin path
pulumi config set --path 'hawk:middlemanAdminGroups[0]' middleman-admin
pulumi up   # short rolling restart of Middleman so the env var lands

# 2. Create the Cognito group and add yourself to it
scripts/dev/manage-cognito-groups.sh <stack> create middleman-admin
scripts/dev/manage-cognito-groups.sh <stack> add-user middleman-admin you@example.com

# 3. Keep your model access: joining any group forfeits the `hawk:defaultPermissions`
#    fallback, so add yourself to every group it names too (default: just
#    `model-access-public`). Skip `create` if the group exists — `... <stack> list`
#    shows what's there.
scripts/dev/manage-cognito-groups.sh <stack> create model-access-public
scripts/dev/manage-cognito-groups.sh <stack> add-user model-access-public you@example.com

# 4. Re-authenticate so the new groups appear in your token
hawk login

Joining middleman-admin drops your default model access

hawk:defaultPermissions only applies to tokens with no group claims at all, so adding yourself to middleman-admin forfeits that fallback — hawk models prints nothing and hawk eval-set fails with Middleman error: Forbidden (an opaque 403, since Middleman's own admin bypass doesn't apply to model-group checks).

Adding and managing models¶

hawk proxy models add claude-haiku-4-5 \
    --group model-access-public \
    --config '{"lab":"anthropic","danger_name":"claude-haiku-4-5","are_details_secret":false,"dead":false,"vision":true,"request_timeout_minutes":30,"is_zdr":false,"raw_cot":false}'

hawk proxy models list                       # show configured models
hawk proxy models get claude-haiku-4-5       # show one model
hawk proxy models update claude-haiku-4-5 --config '{...}'
hawk proxy models deactivate <name>          # disable without deleting
hawk proxy models reload                     # force cache reload
hawk proxy secrets list                      # list configured provider keys

A model's lab must name an active row in the middleman.lab table (Labs and the key-host allowlist); creating or updating a model on any other lab fails with a 400 UnknownLab. The seeded labs include anthropic, openai, openai-chat, gemini-vertex-chat, gemini-developer-api, vertex-serverless, deepseek, mistral, xai, and more — see the seed in the middleman_lab_table migration under hawk/hawk/core/db/alembic/versions/. An update to a model whose lab has no row must also move it to a lab that has one. A model a provider catalog lists under the same name (e.g. openrouter/...) is served on the catalog's lab whatever its config names, so creating, updating or activating it also needs that lab to be active.

hawk proxy models covers the database only

list and get read the model configs stored in the database, not the models Middleman autodiscovers from your provider keys — so get gpt-4o-mini 404s even when that model is callable. Use hawk models for the full catalog you have access to.

Recording zero-data-retention status¶

A model config carries is_zdr: true when requests to that model are covered by a zero-data-retention agreement with the provider, false when they aren't. Middleman doesn't look this up: an admin must set it when adding a model, e.g. hawk proxy models add <name> --group <group> --config '{"lab": "anthropic-chat", "is_zdr": false, "raw_cot": false}', and can change it later with hawk proxy models update <name> --config '{"is_zdr": true}'. Models added before it was required, and routes Middleman discovers itself, read as false. The status appears in the hawk models table and -v output (including for models whose details are secret), and hawk models --zdr true (or false) lists just the models recorded as ZDR (or not).

Some providers do report retention settings through admin APIs — OpenAI's organization and project data-retention endpoints, Anthropic's compliance settings API, and OpenRouter's list of ZDR endpoints — and they're a good source when deciding what to record. They describe an organization, project or endpoint rather than a model, though, so effective coverage still depends on which credentials, workspace and features a model config uses. Status recorded on a third-party route (e.g. openrouter/openai/gpt-4o) never carries over to the direct model Middleman discovers from it (gpt-4o), and a configured model a provider catalog lists under another lab reads as false, since its recorded status covered the configured route (Adding and managing models).

Recording raw chain-of-thought status¶

A model config may also carry raw_cot: true when that model's responses include its raw, unsummarized chain-of-thought (reasoning), false when they don't — for example, when the provider returns only a summary or omits reasoning entirely. It is a label, not a switch: setting it changes nothing about the request Middleman sends. Like is_zdr, an admin must set it when adding a model and can change it later, e.g. hawk proxy models update <name> --config '{"raw_cot": true}'; an open-weight model that always returns its reasoning is simply raw_cot: true. A model added before it was required, or a route Middleman discovers itself, may have no recorded value and reports null, meaning unknown: unlike is_zdr, don't read it as false, since assuming either way can be wrong in a costly direction (mishandling transcripts that do hold raw reasoning, or treating summaries as raw). Once set, the value can be changed but not reset to null. The status appears in the hawk models table and -v output (including for models whose details are secret), and hawk models --raw-cot true|false|unknown lists just the models with that status (unknown finds the ones still to be labelled).

Whether reasoning comes back raw often depends on the route rather than the model: some providers only return it under a particular account or organization, or when a request option is set. The usual pattern is then to configure separate Middleman models for the same underlying model — e.g. <model>-with-cot and <model>-no-cot — each with its own raw_cot value, so the model name recorded in an eval log shows which one was used. The flag describes the route's current behaviour, so if a route's behaviour changes, configure a new model name rather than flipping the flag on the existing one. As with is_zdr, a value recorded on a third-party route never carries over to the direct model Middleman discovers from it.

Recording the serving provider¶

A model config may carry provider: the company Middleman sends that model's requests to, i.e. the company that serves the request and receives the data. Claude called through Anthropic's API is anthropic, but Claude called through Vertex is google; Llama called through Together is together. A model called through OpenRouter is openrouter, since OpenRouter is the company Middleman sends the data to, even though it forwards the request onward. This is distinct from inspect's model-name provider prefixes (e.g. openai-api/...). It's meant for telling which company received data, e.g. to avoid feeding outputs from one provider's route into another provider's route, so it only accepts one lowercase company name per provider: amazon (Bedrock), anthropic, cerebras, deepinfra, deepseek, fireworks, google (Gemini API, Vertex AI), groq, hyperbolic, meta, microsoft (Azure), mistral, openai, openrouter, together, xai. Any other value, including a different capitalization, is rejected; the list lives in PROVIDERS in middleman/src/middleman/model_info.py, so add a company there when you route to a new one.

The provider describes the route, not the model: like is_zdr, a value recorded on a third-party route (e.g. openrouter/openai/gpt-4o) never carries over to the direct model Middleman discovers from it (gpt-4o). Routes Middleman discovers itself have no config row, so it fills them in: catalog entries from OpenRouter, Fireworks and Together get that company, and a model confirmed against OpenAI's or Anthropic's own model list gets that lab. If a catalog entry shares a configured model's name, the catalog's lab and provider both win, so the two always describe the same route. A configured model with no recorded provider gets its lab's default, because the lab decides which company receives the requests: anthropic* labs are anthropic, openai* labs are openai, the seeded gemini-* and vertex-serverless labs are google, and the other labs are their own company. A model on a lab with no middleman.lab row (e.g. dummy, dummy-chat, gemini, gemini-public, vertex) has no provider. The defaults are the provider column of the middleman.lab table. A recorded value always wins, so only record one when the lab's default is wrong for that model, e.g. hawk proxy models update <name> --config '{"provider": "<company>"}'; a wrong value makes Hawk treat the model as served by that company. Once set, the value can be changed but not reset to null.

It appears in the hawk models table and -v output, except for models whose details are secret, where it's hidden like lab (a pre-release model is usually served by the lab that made it); the admin API still shows it. A model with no provider (one on a lab with no middleman.lab row, e.g. dummy, gemini, vertex) reports null, meaning unknown: consumers comparing providers should fail closed rather than assume an unknown provider matches, or differs from, another. Middleman's /model_groups endpoint returns a providers map alongside labs for the queried models, including models whose details are secret. Models with no provider are left out of that map, so treat a missing entry as unknown. The admin API shows only what was recorded, not the lab default.

Recording who may read a model's transcripts¶

Hawk stops a reader model (a scanner, or a transcript-search model) from reading the non-public transcripts a source model produced unless the reader is served by the same provider (How the cross-lab rule decides). Where a lab has agreed to something else, record it on the models; Middleman stores and serves both fields and Hawk applies them:

read_policy, on a source model, says which readers may read its transcripts: {"mode": "provider" | "any" | "tags", "tags": [...], "zdr_only": false}. provider (the default) admits readers whose provider matches the source's, any admits every reader, and tags admits readers holding at least one of its tags (which mode tags requires; the other modes ignore them). zdr_only: true additionally requires the reader to be recorded as is_zdr (Recording zero-data-retention status). Absent or null is the default, {"mode": "provider", "tags": [], "zdr_only": false}.
reader_tags, on a reader model, lists the tags it holds. Absent, null or [] is none.

Set policies only once Hawk applies them

Hawk applies these fields only from the version that reads read_access from Middleman's /model_groups. An older Hawk ignores them and judges every source as mode provider, silently: a policy meant to restrict readers (mode tags, or zdr_only) restricts nothing, and one meant to admit them admits nobody new. Don't set a non-default policy until every Hawk deployment that uses this Middleman applies them.

Tags are lowercase letters, digits and hyphens, starting with a letter or digit (^[a-z0-9][a-z0-9-]*$); they are de-duplicated and sorted when written. Make tag names opaque (e.g. reader-tier-2): never name a model, a lab or an agreement in one. They aren't secret: /model_groups returns a model's policy tags and reader tags to every caller who can query that model there (anyone with a grant on its model group, or the middleman:query_model_groups permission), so the tags on a reader in model-access-public reach every user, and a tag named after a secret model or deal would reveal it. Set them with e.g. hawk proxy models update <source> --config '{"read_policy": {"mode": "tags", "tags": ["approved"]}}' and hawk proxy models update <reader> --config '{"reader_tags": ["approved"]}', and clear either by setting it to null. A policy in an update replaces the stored one whole, so fields it leaves out take their defaults. A model in model-access-public can't have a non-default policy (any mode other than provider, or zdr_only: true), since Hawk doesn't apply policies to public models' transcripts; that includes an update moving a model with one into that group. Every change to read_policy, reader_tags or is_zdr (which grants a reader zdr_only sources) is logged as admin.model.read_access_changed with the model, the admin and the values before and after.

Both describe a route, like is_zdr: a value recorded on a third-party route never carries over to the direct model Middleman discovers from it, and neither may be set in base model info. Likewise, if a provider catalog lists a configured model under another lab (so it's served there, Adding and managing models), the model loads without its configured reader_tags and is_zdr, since they were approved for the configured route, and a non-default read_policy blocks every reader (falling back to the default could loosen a restrictive one) until an admin approves the model for the new route. A default policy is simply dropped. Middleman logs model_config.route_approvals_dropped when this drops any. Neither read_policy nor reader_tags appears in public model info (hawk models). Middleman's /model_groups endpoint returns them in a read_access map, alongside providers and with the same access check, including for models whose details are secret; a model whose policy is the default, with no reader tags and not recorded as ZDR, is left out, so treat a missing entry as the default. Each entry is {"policy": {"mode": ..., "tags": [...], "zdr_only": ...}, "reader_tags": [...], "is_zdr": ...}: policy has the same shape as the stored read_policy and describes the model as a source, and the other fields describe it as a reader. Every field inside policy restricts who may read the model's transcripts, so a field added there later keeps that meaning and a client that sees one it doesn't know set to a non-empty or true value must fail closed for that source; a client ignores top-level fields it doesn't know.

A malformed value in a hand-edited config row fails closed: a policy Middleman can't read (not an object, an unknown key or mode, tags not a list, zdr_only not a boolean) becomes mode tags with no tags, so no reader can read the model's transcripts until it's fixed; invalid tags are dropped, and mode tags with no valid tags (including an empty list) leaves nobody able to read, which is also reported; and a reader_tags that isn't a list, or its invalid entries, are dropped. Middleman logs an error for each, and the admin API shows the value as enforced along with the fault in read_policy_error or reader_tags_error, so it can be found and fixed.

Per-model Anthropic account routing¶

A model config may carry an anthropic_account field naming one of the profiles configured via hawk:middlemanAnthropicProfiles (or hawk:middlemanAnthropicProfilesJson). Requests for that model then authenticate to Anthropic with that profile's workload-identity-federation (OIDC client-credentials) setup instead of the shared ANTHROPIC_API_KEY; unknown account names are rejected when the model is created or updated. Most deployments don't need this — setting ANTHROPIC_API_KEY covers the common case.

To run without static Anthropic keys, set hawk:middlemanAnthropicDefaultProfile to one of those profiles. Every Anthropic model without its own anthropic_account (including models found by discovery) then uses it on the passthrough API, and model discovery authenticates with it too. Predeployment models (anthropic-chat-predeployment) never use the default: without an account of their own they keep the predeployment keys. A model routed to a profile never falls back to ANTHROPIC_API_KEY when that profile fails. Give the default profile a high_priority_profile (below) to keep high-priority requests on separate capacity. Priority quota groups keep matching these models with baseline_profile: null, and a group's common_profile can't be the default profile.

Dev stacks and credential profiles¶

Dev stacks run in stg's AWS account, so for every provider (Anthropic, OpenAI, OpenRouter and Google) they take stg's hawk:middleman<Provider>Profiles and hawk:middleman<Provider>DefaultProfile unless they set their own profiles, in which case stg's default doesn't apply. A default profile set locally always wins: naming another inherited profile switches to it, and "" turns the inherited default off so the provider's static keys serve instead. An inherited aws_sts profile presents the dev stack's own task role (dev-<name>-middleman-tasks), so the provider's federation rule or identity provider must admit that role as well as stg's.

Requiring `anthropic-beta` flags¶

A model on a lab served by the Anthropic API (as seeded, anthropic, anthropic-chat and anthropic-chat-predeployment) may carry required_anthropic_betas, a list of flags every request to it must send in the anthropic-beta header. This suits a route that exists to use a beta feature — for example, two model configs pointing at the same Anthropic model through different anthropic_account orgs, where only one org has the feature enabled.

Middleman validates the header it forwards and never adds a flag to it. A request to /anthropic/v1/messages missing any required flag gets a 400 naming the missing flags; flags are matched exactly (case-sensitive), in any order, alongside any others. If the header is sent more than once, the copies are combined into one comma-separated value, as Anthropic reads them, so the flags may appear in any copy.
/anthropic/v1/messages/count_tokens is exempt.
A malformed value in a hand-edited config row (or one on a non-Anthropic lab) fails closed: Middleman logs an error and refuses every request to that model, on every route, with a 500 until it's fixed. The admin API still reads such a row and reports the fault in required_anthropic_betas_error, so it can be found and cleared.
The list isn't included in public model info (hawk models), since flag names can reveal unreleased features.

Set it with hawk proxy models update <name> --config '{"required_anthropic_betas": ["example-beta-2025-01-01"]}' and clear it by setting it to null. Entries are trimmed and de-duplicated; an empty list, a blank entry, or one containing a comma, whitespace, a control character or anything other than visible ASCII is rejected, as is setting the field on (or moving a model with it to) a non-Anthropic lab, or on a model a provider catalog serves on one.

Per-model OpenAI account routing¶

The OpenAI counterpart: a model config may carry an openai_account field naming a profile from hawk:middlemanOpenaiProfiles (or hawk:middlemanOpenaiProfilesJson). OpenAI WIF profiles authenticate as the middleman ECS task role (assertion_source: "aws_sts"), so they require hawk:middlemanStsWebIdentityEnabled: "true"; the resulting task-role policy allows both the Anthropic and OpenAI token audiences. When creating the identity provider in the OpenAI dashboard, set its Audience to exactly https://api.openai.com/v1: OpenAI lets you choose this value, but middleman always requests that audience from STS and the task-role policy allows only it. Setting OPENAI_API_KEY remains the common case.

openai_account is for OpenAI accounts and profiles. Whatever the profile's type, its credential is checked against the key-host allowlist as OPENAI_ACCOUNT, which by default only reaches api.openai.com, so a model create, PATCH or activate is refused with a 400 OpenAIAccountHostNotAllowed if the model's lab (or a lab a provider catalog serves it on) is an OpenAI-compatible lab whose host isn't allowed for OPENAI_ACCOUNT, and a lab PATCH or activate that would leave an active model's lab refusing its openai_account this way (or the WIF way below) is refused with a 409 LabInUseByOpenAIAccount (Managing labs). Labs keyed by OPENROUTER_API_KEY ignore the field and aren't checked. For a per-model account at another OpenAI-compatible provider, add a lab row of its own whose api_key_name is that account's key (Managing labs) and put the model on it, rather than allowing the provider's host for OPENAI_ACCOUNT in hawk:middlemanProviderKeyHosts, which would let any profile's credential reach it.

To run without static OpenAI keys, set hawk:middlemanOpenaiDefaultProfile to one of those profiles. Models on the labs that use OPENAI_API_KEY (openai-chat, openai-responses, and the legacy-completions labs) with no openai_account of their own then use it on the passthrough API, and so do batch file uploads and model discovery. The *-with-data-retention labs never use the default: they keep OPENAI_API_KEY_WITH_DATA_RETENTION, which belongs to a different organization, and a WIF profile may not be pinned on them (or on a third-party OpenAI-compatible lab that uses a static key, or on a model a provider catalog serves on one) via openai_account — that token is scoped to this org. OpenAI priority overflow routes still use the static keys, and saving a priority config with an OpenAI group requires them, so keep OPENAI_API_KEY until those move too. Give the default profile a high_priority_profile to keep a separate high-priority lane.

OpenAI Decisions¶

POST /openai/v1/decisions uses the existing OpenAI passthrough and model aliases. To track Decisions usage separately, add another alias for the same upstream model. For a model using the normal OpenAI credentials:

hawk proxy models add gpt-6-luna-decisions --group model-access-public \
  --config '{"lab":"openai-responses","danger_name":"gpt-6-luna","are_details_secret":false,"is_zdr":false,"raw_cot":false}'

With an OpenAI Python client configured for Middleman (SDK 3.26.0 or later):

decision = client.decisions.create(
    model="gpt-6-luna-decisions",
    input="The package arrived with a dent.",
    questions=[{"type": "predicate", "instructions": "Is the package damaged?"}],
)

See OpenAI's Decisions guide for API details.

OpenRouter credential routing¶

OpenRouter models (those on a lab whose key is OPENROUTER_API_KEY, like the seeded openrouter lab) authenticate with that key unless hawk:middlemanOpenrouterDefaultProfile names a profile from hawk:middlemanOpenrouterProfiles (or hawk:middlemanOpenrouterProfilesJson). Every OpenRouter request then uses that profile, and a model's openai_account is ignored. OpenRouter WIF profiles authenticate as the middleman ECS task role (assertion_source: "aws_sts"), so they require hawk:middlemanStsWebIdentityEnabled: "true". In OpenRouter (Business and Enterprise plans), register the account's STS issuer, then create a federation policy with Subject set to the task role ARN and Audience set to exactly https://openrouter.ai/api/v1. The policy acts as one of the org's workspace API keys, so usage bills to that key, and disabling the key cuts off access immediately. Set the profile's federation_policy_id to the policy's ID. As with the other providers, a failing profile returns an error instead of falling back to the key.

HIGH-priority profile routing¶

Anthropic and OpenAI account profiles can set high_priority_profile to another profile name in the same provider's configuration. Requests with X-Middleman-Priority: high use that profile's credentials; other requests keep using the model's configured profile. The target can use either an API key or a WIF authentication method supported by that provider. For Anthropic WIF, configure the target profile with the intended HIGH workspace's workspace_id.

For example, this Pulumi configuration routes HIGH requests for models using anthropic_account: regular to the high profile:

hawk:middlemanAnthropicProfiles:
  regular:
    auth_type: api_key
    api_key_env: ANTHROPIC_API_KEY
    high_priority_profile: high
  high:
    auth_type: api_key
    api_key_env: ANTHROPIC_HIGH_PRIORITY_API_KEY

Selection happens once: if A points to B and B points to C, a HIGH request starting at A uses B, while one starting at B uses C. The selected target's own high_priority_profile and api_key_env_high are not applied again. Self-references and cycles therefore do not recurse. Referenced names must exist when profiles are loaded; a missing target is a configuration error.

Without high_priority_profile, existing behavior is preserved: API-key profiles use api_key_env_high for HIGH requests when configured, and WIF profiles use their normal credentials. An explicit named HIGH profile takes precedence over the source's api_key_env_high. WIF tokens, refresh locks, and cooldowns belong to the selected profile, so multiple sources selecting the same target share its cached token.

Model group naming

Model groups use the prefix model-access-<name> (e.g. model-access-public), and the user's JWT must carry a matching group. For Cognito users without explicit group claims, hawk:defaultPermissions provides the fallback (default "model-access-public"). The admin group itself must not use the model-access- prefix and must not overlap with defaultPermissions — Middleman refuses to start if either constraint is violated.

Managing labs¶

Labs (Labs and the key-host allowlist) are managed with hawk proxy labs, which needs Middleman admin like hawk proxy models:

hawk proxy labs list                         # active labs (--status inactive|all, --format table)
hawk proxy labs get deepseek                 # one lab, active or not

# Add an OpenAI-compatible lab. Its key must be allowed for its host; see below.
# --config takes JSON or @path/to/file.json; the name comes from the argument.
hawk proxy labs add acme --config '{"api": "openai", "base_url": "https://api.acme.example/v1",
  "credential_kind": "api_key", "api_key_name": "ACME_API_KEY"}'

# Change some fields; omitted fields keep their value, null clears a nullable one.
hawk proxy labs update acme --config '{"unsupported_paths": ["/responses/compact"]}'

hawk proxy labs deactivate acme
hawk proxy labs activate acme

Then set the lab's key with hawk proxy secrets set and add models on it with hawk proxy models add.

These wrap Middleman's /admin/labs/ API, which takes the same admin access token as the rest of the admin API: GET /admin/labs/ (?status=active|inactive|all), GET /admin/labs/{name}, POST /admin/labs/ (201), PATCH /admin/labs/{name}, and POST /admin/labs/{name}/deactivate and /activate. Errors come back with an error_name (e.g. KeyHostNotAllowed, LabInUse, LabNameImmutable), which the CLI prints along with the status.

With the shared Valkey cache configured, every write is broadcast and each worker on every replica reloads within moments. Without Valkey, workers keep the previous labs until their hourly refresh, so restart Middleman to apply a change immediately.

Field	Rules
`name`	Required; lowercase letters, digits, `.`, `_` and `-`, starting with a letter or digit; at most 64 characters. Can't change after creation: model configs reference the lab by name (a PATCH with a different `name` is a 400 `LabNameImmutable`).
`api`	Required; the passthrough API serving the lab: `anthropic`, `openai` (chat completions and responses), `openai-legacy-completions`, `gemini-vertex` or `gemini-developer`.
`base_url`	The API root the request path is appended to. Required for `openai`, `openai-legacy-completions` and `gemini-vertex`; must be null for `anthropic` and `gemini-developer`, whose routes use fixed hosts. Must be `https://` with a DNS host name, not an IP address (so the host can't be a literal such as `169.254.169.254` or `[::1]`, or end in a numeric label such as `127.1`), and have no `user@`, query string, fragment or trailing `/`. The only placeholder is `{gcp_project}` (filled in with the GCP project per request), and only in the path, so the host is known when the lab is written.
`credential_kind`	`api_key` or `gcloud_token` (the GCP service-account token). Required for `openai` and `openai-legacy-completions`; must be null for the other APIs (`gemini-vertex` always sends the GCP token).
`api_key_name`, `high_priority_api_key_name`	Provider-key names (`[A-Z0-9_]+`, at most 128 characters, as set with `hawk proxy secrets set`), only with `credential_kind: api_key`, which requires `api_key_name`. The high-priority key, if set and present, is sent for `X-Middleman-Priority: high` requests. A lab whose `api_key_name` is `OPENAI_API_KEY` uses the OpenAI default credential profile (`hawk:middlemanOpenaiDefaultProfile`) when one is configured, whatever its high-priority key; otherwise the lab's own keys are used. OpenAI priority routing applies to `openai` labs whose `api_key_name` is `OPENAI_API_KEY` or `OPENAI_API_KEY_WITH_DATA_RETENTION`, and its overflow route sends that key family's high-priority key (`OPENAI_HIGH_PRIORITY_API_KEY` or `OPENAI_HIGH_PRIORITY_API_KEY_WITH_DATA_RETENTION`) even if the lab names a different one or none.
`unsupported_paths`	Request paths, relative to `base_url`, the upstream doesn't implement (Middleman answers them with a 404), e.g. `["/responses/compact"]`. `openai` labs only; each starts with `/`, is not `/` alone, uses only letters, digits and `._~/-`, and is at most 256 characters, with no repeats. Default `[]`.
`provider`	The provider a model on the lab gets when it records none; null or one of the allowed provider values (Recording the serving provider).
`is_chat`	Whether the lab's models take chat messages rather than a plain text prompt. Default `true`.

A lab breaking these rules is refused with a 422, and a PATCH is checked on the lab as it would be after the change, so moving a lab to another api means clearing the fields that API doesn't take in the same request. Moving an anthropic lab to another api is refused with a 409 LabInUse while any active model on it sets required_anthropic_betas, which only the Anthropic API can honour; the error names up to 10 of them. A PATCH or activate is refused with a 409 LabInUseByOpenAIAccount if any active model on the lab pins an openai_account the lab, as it would stand, must not receive, by the rules a model write checks (Per-model OpenAI account routing): moving openai-chat to another vendor's base_url and key while a model on it pins an API-key profile (its credential, sent as OPENAI_ACCOUNT, isn't allowed for the new host, so every request would fail with a 500), or onto OPENAI_API_KEY_WITH_DATA_RETENTION while one pins a WIF profile (a token scoped to the other organization). The error names up to 10 of those models and why each account is refused; clear their openai_account, move them, or deactivate them first. Every such model counts, not only those the change itself breaks, so after an allowlist change a lab takes no write until its models are fixed. Both checks count models discovered from provider catalogs on the lab as well as those whose config names it (see Deactivating), reading whether they require betas or pin an account from their stored config. Responses carry every field plus is_active.

Key-host allowlist. A create, PATCH or activate is refused with a 400 KeyHostNotAllowed if the lab would send any credential to a host the allowlist doesn't permit for it. That covers the lab's api_key_name and high_priority_api_key_name, the GCP token for gcloud_token and gemini-vertex labs, the OpenAI default credential profile for labs whose api_key_name is OPENAI_API_KEY, the OpenRouter credential profile for labs whose api_key_name is OPENROUTER_API_KEY, and, for labs in OpenAI priority routing, both keys of that key family (e.g. OPENAI_API_KEY_WITH_DATA_RETENTION and OPENAI_HIGH_PRIORITY_API_KEY_WITH_DATA_RETENTION), whatever the lab's high_priority_api_key_name. A model's own openai_account, which is for OpenAI accounts and profiles, is checked when the model is written: a model create, PATCH or activate is refused with a 400 OpenAIAccountHostNotAllowed if its lab's host isn't allowed for OPENAI_ACCOUNT (Per-model OpenAI account routing), so a per-model account at another OpenAI-compatible provider belongs on a lab row of its own. The error names the credential and the host, never a key value. Passthrough re-checks before every request (Labs and the key-host allowlist), so a lab edited in the database, or one a narrower allowlist no longer permits, fails with a 500 rather than leaking a key.

Deactivating. deactivate is the soft delete: the lab stops loading, so no route serves it. It's refused with a 409 LabInUse, naming up to 10 of them, while any active model uses the lab; deactivate those models or move them to another lab first. That includes models discovered from provider catalogs, which no active model config names (a deactivated model a catalog still lists counts too): OpenRouter, Fireworks and Together models, and the public OpenAI and Anthropic models on openai-chat and anthropic-chat. They can't be moved or deactivated, so a lab that serves them stays active while a catalog lists them. Models can't be created on, moved to, or re-activated on a deactivated lab (400 UnknownLab). activate re-checks the stored lab against today's rules and allowlist first, so a lab written under older rules, or edited in the database, has to be fixed with a PATCH before it's re-enabled. Labs are never deleted, so a name is never reused.

Admin gate covers provider keys too

The admin gate covers model management (/admin/models/), lab management (/admin/labs/) and provider-key rotation (/admin/secrets/provider-keys). The key-host allowlist is what stops an admin from pointing a lab at their own host to read a provider key, so changing hawk:middlemanProviderKeyHosts deserves the same review as granting admin. Anyone who can run cognito-idp:AdminAddUserToGroup on your user pool can grant admin transitively — treat that IAM permission as equivalent to Middleman admin in policy reviews.

Revoking admin: removing a user from the admin Cognito group does not immediately revoke their existing access token; it stays admin until its TTL expires (1 hour by default for Cognito access tokens). Refresh tokens (30 days default) mint fresh access tokens without the removed claim, so the practical revocation window is the access-token TTL. For immediate revocation, run aws cognito-idp admin-user-global-sign-out --user-pool-id <pool> --username <email> — this invalidates all of that user's tokens.

Usage attribution by user¶

The user dimension on Middleman's token-usage metrics, and the user_email field of its traffic log, carry the caller's email address: Middleman reads the claim named by hawk:oidcEmailField (default email), the same setting the Hawk API uses to identify the person. A token without that claim — a machine client's, or an IdP that never emits one — is attributed to its JWT sub instead, so nothing is dropped; the traffic log always records user_id (the sub) as well, and that is the field to join on. If your IdP puts the address in a different claim, point hawk:oidcEmailField at it and the API, relay and Middleman all follow. On a Hawk-provisioned Cognito stack the access token carries no email claim, so callers stay attributed by sub; getting an email there needs a pre-token-generation Lambda that adds the claim. The dimension is recorded at request time, so per-user series keyed on an opaque subject end at the deploy that enabled this and new ones begin under the email.

Usage attribution by team¶

Middleman emits its token-usage CloudWatch metrics with a team dimension alongside the per-user one, and the viewer's usage page can group by it (/usage?group_by=team). The team comes from the user's access token: Middleman reads the claim named by hawk:oidcTeamsClaim (default teams), strips hawk:oidcTeamsGroupPrefix (default team-) from each value, and uses the result as the team name. A token whose claim is missing or empty is attributed to unassigned; a user in several teams is attributed to the sorted, +-joined combination (e.g. alignment+infra) rather than double-counted. A group whose stripped name is unassigned or contains + would collide with those labels, so Middleman drops it and logs auth.invalid_team_name at error level; rename the group.

The convention is to name IdP groups team-<name> and have the IdP emit them in a teams claim on the access token:

Okta — on the authorization server, add a Groups claim named teams to the access token with filter Starts with team-.
Other IdPs — any provider works as long as the access token carries a string-list claim of group names; point hawk:oidcTeamsClaim at it and set hawk:oidcTeamsGroupPrefix to whatever prefix your groups use (or "" to keep the full group name).
Set hawk:oidcTeamsClaim: "" to disable team attribution entirely; every request is then attributed to unassigned.

The dimension is recorded at request time, so team history starts from the deploy that enabled it — there is no backfill of earlier usage, and moving a user between groups only affects requests made after their next token refresh.

Usage attribution by job and task¶

The usage API can also group by job (eval set or scan id, GET /usage?group_by=job) and by Inspect task name (group_by=task). Those groupings are far too high-cardinality for a CloudWatch metric dimension, so the Hawk API answers them from Middleman's traffic log instead: Logs Insights queries over the <env>/middleman/traffic log group, reading each record's public_name, token counts and the x-hawk-job-id / x-hawk-job-type / x-inspect-task-name correlation headers the runner stamps on model requests.

This needs the traffic log at level summary or full (hawk:middlemanTrafficLogLevel; summary is the default). When it is on, the stack sets HAWK_API_MIDDLEMAN_TRAFFIC_LOG_GROUP on the API and grants its task role logs:StartQuery on that log group; at level off the job and task groupings return no data. Token counts are present in the traffic log only for requests since 2026-09-18, and history is bounded by the log group's retention. See Token Usage for the endpoints, label semantics (direct, scan, unknown, other) and caveats.

GitHub Actions workload identity¶

A stack can let named GitHub Actions workflows call Middleman with GitHub's short-lived OIDC token instead of a stored key. It is off until you list a workflow; deploy a Middleman that supports it before adding entries:

hawk:middlemanGitHubActionsWorkloads:
  - name: "pr-review"                   # keep stable: usage and quotas are keyed on it
    audience: "hawk-review-panel"
    repositoryId: "123456789"           # numeric GitHub IDs, quoted
    repositoryOwnerId: "987654321"
    workflowRef: "your-org/hawk/.github/workflows/panel-review.yml@refs/heads/main"
    environment: "panel-review"
    modelGroups: ["model-access-public"]  # existing groups; never `middleman:` permissions
    teams: ["platform"]                 # optional; otherwise usage is "unassigned"

A token gets in only if exactly one entry matches its audience, repository, owner, workflow ref and environment; it then gets that entry's model groups, and never admin. Add an entry per workflow. Entries are per stack, and dev stacks don't inherit them. To configure Middleman directly, see the workloads rules in middleman.yaml.example.

The workflow must:

use the named environment, restricted to the trusted branch, and have id-token: write;
request a token with the configured audience and send it as Authorization: Bearer <token> (OpenAI endpoints) or x-api-key: <token> (Anthropic endpoints);
fetch a fresh token before the last expires, within minutes, for example through Claude Code's apiKeyHelper or a Codex provider's auth command;
reach Middleman, for a private deployment through Tailscale or similar.

To keep it safe:

A pull_request run never matches, so a workflow edited in a PR can't get a token. Every other run of the trusted branch's workflow does, including pull_request_target, issue_comment, issues and workflow_run, which anyone can start on a public repository. Unless the workflow runs only on push, workflow_dispatch or schedule, check who started the run before any job with id-token: write.
Every step of a job with id-token: write can mint a token, so run PR code in a separate job without it.

To revoke a workflow, remove its entry and deploy. To stop it sooner, disable the workflow and cancel its running jobs. Middleman checks tokens offline, so one already issued keeps working until it expires, within minutes.

Deploying Changes¶

Middleman runs on ECS Fargate. Deployments are triggered by pushing to the main branch, which builds a new Docker image and updates the ECS service via CI/CD.

Running Locally¶

cd middleman
# Add API keys to .env (see example.env)
docker compose up --build

Testing the Passthrough API¶

uv run scripts/exercise_passthrough.py --help

This script tests the passthrough API against multiple providers (Anthropic, OpenAI, OpenRouter).