LLM Proxy (Middleman)¶
Middleman is Hawk's built-in LLM proxy. It runs on ECS Fargate and routes model API calls to providers (OpenAI, Anthropic, Google Vertex, DeepSeek, Fireworks, and more) with automatic token refresh and access control.
How It Works¶
When evaluations run on the cluster, Inspect AI sends model API calls through Middleman instead of directly to providers. Middleman:
- Authenticates the request using the runner's scoped credentials
- Routes the request to the correct provider API
- Handles token refresh and retries
- Enforces model group permissions
Callers use each provider's own API format through the passthrough routes (/anthropic/v1/messages, /openai/v1/chat/completions, /openai/v1/responses, /openai/v1/decisions, /gemini/...). The older unified /completions, /count_prompt_tokens and /embeddings APIs have been removed; /embeddings has no passthrough replacement. Models on labs that only the unified API served (gemini, gemini-public, vertex, dummy and dummy-chat) still load and appear in model listings, but no route can call them.
Setting Up API Keys¶
Provider API keys are set through Middleman's admin API (Granting admin below), wrapped by hawk proxy secrets set. The helper script takes KEY=VALUE pairs; the target deployment is selected by HAWK_MIDDLEMAN_URL, so run hawk login first:
Set multiple keys at once:
Setting a key rediscovers models straight away. The response reports the resulting catalog size and a per-provider outcome — ok, cached (upstream failed, serving the last good list), failed, no_key, not_run (cleared, and the reload has not refetched it yet), or stale (the reload failed before refetching it) — so you can tell whether the key took effect without reading the proxy's logs. hawk proxy models discovery shows the same without triggering a fetch; hawk proxy models reload forces one. A slow provider can outlast the response budget; the write still stands and the rediscovery keeps running, so re-check it with hawk proxy models discovery. If the reload fails, the write stands too; retry with hawk proxy models reload.
Supported Providers¶
OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, GOOGLE_APPLICATION_CREDENTIALS_JSON, DEEPINFRA_TOKEN, DEEPSEEK_API_KEY, FIREWORKS_API_KEY, HYPERBOLIC_API_KEY, META_API_KEY, MISTRAL_API_KEY, OPENROUTER_API_KEY, TOGETHER_API_KEY, XAI_API_KEY, plus OPENAI_HIGH_PRIORITY_API_KEY and ANTHROPIC_HIGH_PRIORITY_API_KEY.
High-priority requests use the dedicated high-priority key when present, otherwise the matching ordinary key. This also applies within Anthropic predeployment and OpenAI data-retention credentials; fallback never crosses those credential families. Explicitly configured credential-profile errors still fail.
Vertex-routed models require a GCP project
Models whose lab routes through Vertex AI need a GCP project: set hawk:middlemanGcpProjectForPublicModels (or include a project_id in GOOGLE_APPLICATION_CREDENTIALS_JSON). Deployments with no Vertex-routed models don't need this. If a Vertex-routed model (gemini-vertex-chat, gemini-vertex-chat-global, vertex-serverless) is configured without a project, Middleman fails to start. A model added later, by a reload, logs models.gcp_project_missing instead, and its requests fail with a 500 saying GCP project not configured.
High Availability¶
For opt-in priority allocation, quota auditing and shadow validation, see Priority and per-user allocation.
By default Middleman runs a single replica. Setting hawk:highAvailability: "true" runs multiple replicas across AZs, which requires the shared Valkey cache (hawk:valkeyEnabled: "true" or an external hawk:valkeyUrl) — replicas share their provider-key/model caches and serialize secret writes through Valkey; without it they serve divergent caches and race secret writes. Deploys fail fast if HA is enabled without Valkey. See the configuration reference.
Bypassing the Proxy¶
To use your own API keys instead of Middleman, pass the matching key as a secret, disable the proxy's token refresh, and override that provider's base URL so it points directly upstream. For OpenAI, for example:
Then pass your API key as a secret:
Managing Models¶
Model configurations are stored in the database and organized into model groups for access control — a user must belong to a model's group to use it. A fresh deploy starts with an empty model registry, so hawk eval-set fails with Middleman error: Models not found until you add at least one model.
Models are managed through Middleman's admin API, wrapped by the hawk proxy models CLI commands.
Granting admin¶
The admin API is gated by is_admin=true. There are two ways to grant it:
- Auth0 / Okta: emit the
https://middleman.metr.org/claims/admin: trueJWT claim. Manage admins in your identity provider. - Cognito (default open-source deploy): set
hawk:middlemanAdminGroupsin your Pulumi config to a non-empty list (e.g.["middleman-admin"]) and add users to that Cognito group. The default is empty, so the group-based admin path is opt-in.
For Cognito:
# 1. One-time per stack: opt in to the group-based admin path
pulumi config set --path 'hawk:middlemanAdminGroups[0]' middleman-admin
pulumi up # short rolling restart of Middleman so the env var lands
# 2. Create the Cognito group and add yourself to it
scripts/dev/manage-cognito-groups.sh <stack> create middleman-admin
scripts/dev/manage-cognito-groups.sh <stack> add-user middleman-admin you@example.com
# 3. Keep your model access: joining any group forfeits the `hawk:defaultPermissions`
# fallback, so add yourself to every group it names too (default: just
# `model-access-public`). Skip `create` if the group exists — `... <stack> list`
# shows what's there.
scripts/dev/manage-cognito-groups.sh <stack> create model-access-public
scripts/dev/manage-cognito-groups.sh <stack> add-user model-access-public you@example.com
# 4. Re-authenticate so the new groups appear in your token
hawk login
Joining middleman-admin drops your default model access
hawk:defaultPermissions only applies to tokens with no group claims at all, so adding yourself to middleman-admin forfeits that fallback — hawk models prints nothing and hawk eval-set fails with Middleman error: Forbidden (an opaque 403, since Middleman's own admin bypass doesn't apply to model-group checks).
Adding and managing models¶
hawk proxy models add claude-haiku-4-5 \
--group model-access-public \
--config '{"lab":"anthropic","danger_name":"claude-haiku-4-5","are_details_secret":false,"dead":false,"vision":true,"request_timeout_minutes":30}'
hawk proxy models list # show configured models
hawk proxy models get claude-haiku-4-5 # show one model
hawk proxy models update claude-haiku-4-5 --config '{...}'
hawk proxy models deactivate <name> # disable without deleting
hawk proxy models reload # force cache reload
hawk proxy secrets list # list configured provider keys
Valid lab values include anthropic, openai, openai-chat, gemini-vertex-chat, gemini-developer-api, vertex-serverless, deepseek, mistral, xai, and more — see LABS in middleman/src/middleman/lab_registry.py. Labs that no route serves are accepted only for config compatibility.
hawk proxy models covers the database only
list and get read the model configs stored in the database, not the models Middleman autodiscovers from your provider keys — so get gpt-4o-mini 404s even when that model is callable. Use hawk models for the full catalog you have access to.
Recording zero-data-retention status¶
A model config may carry is_zdr: true when requests to that model are covered by a zero-data-retention agreement with the provider, false when they aren't. Middleman doesn't look this up: an admin records it by hand, e.g. hawk proxy models update <name> --config '{"is_zdr": true}'. A model with no recorded value reports null, which callers should treat as not ZDR; once set, the value can be changed but not reset to null. The status appears in hawk models -v output (including for models whose details are secret), and hawk models --zdr-only lists just the models recorded as ZDR.
Some providers do report retention settings through admin APIs — OpenAI's organization and project data-retention endpoints, Anthropic's compliance settings API, and OpenRouter's list of ZDR endpoints — and they're a good source when deciding what to record. They describe an organization, project or endpoint rather than a model, though, so effective coverage still depends on which credentials, workspace and features a model config uses. Status recorded on a third-party route (e.g. openrouter/openai/gpt-4o) never carries over to the direct model Middleman discovers from it (gpt-4o).
Recording raw chain-of-thought status¶
A model config may also carry raw_cot: true when that model's responses include its raw, unsummarized chain-of-thought (reasoning), false when they don't — for example, when the provider returns only a summary or omits reasoning entirely. It is a label, not a switch: setting it changes nothing about the request Middleman sends. Like is_zdr, an admin records it by hand, e.g. hawk proxy models update <name> --config '{"raw_cot": true}'; an open-weight model that always returns its reasoning is simply raw_cot: true. A model with no recorded value reports null, meaning unknown: unlike is_zdr, don't read it as false, since assuming either way can be wrong in a costly direction (mishandling transcripts that do hold raw reasoning, or treating summaries as raw). Once set, the value can be changed but not reset to null. The status appears in hawk models -v output (including for models whose details are secret), and hawk models --raw-cot-only lists just the models recorded as returning raw CoT.
Whether reasoning comes back raw often depends on the route rather than the model: some providers only return it under a particular account or organization, or when a request option is set. The usual pattern is then to configure separate Middleman models for the same underlying model — e.g. <model>-with-cot and <model>-no-cot — each with its own raw_cot value, so the model name recorded in an eval log shows which one was used. As with is_zdr, a value recorded on a third-party route never carries over to the direct model Middleman discovers from it.
Recording the serving provider¶
A model config may carry provider: the company Middleman sends that model's requests to, i.e. the company that serves the request and receives the data. Claude called through Anthropic's API is anthropic, but Claude called through Vertex is google; Llama called through Together is together. A model called through OpenRouter is openrouter, since OpenRouter is the company Middleman sends the data to, even though it forwards the request onward. This is distinct from inspect's model-name provider prefixes (e.g. openai-api/...). It's meant for telling which company received data, e.g. to avoid feeding outputs from one provider's route into another provider's route, so it only accepts one lowercase company name per provider: amazon (Bedrock), anthropic, cerebras, deepinfra, deepseek, fireworks, google (Gemini API, Vertex AI), groq, hyperbolic, meta, microsoft (Azure), mistral, openai, openrouter, together, xai. Any other value, including a different capitalization, is rejected; the list lives in PROVIDERS in middleman/src/middleman/model_info.py, so add a company there when you route to a new one.
The provider describes the route, not the model: like is_zdr, a value recorded on a third-party route (e.g. openrouter/openai/gpt-4o) never carries over to the direct model Middleman discovers from it (gpt-4o). Routes Middleman discovers itself have no config row, so it fills them in: catalog entries from OpenRouter, Fireworks and Together get that company, and a model confirmed against OpenAI's or Anthropic's own model list gets that lab. If a catalog entry shares a configured model's name, the catalog's lab and provider both win, so the two always describe the same route. A configured model with no recorded provider gets its lab's default, because the lab decides which company receives the requests: anthropic* labs are anthropic, openai* labs are openai, the gemini* and vertex* labs are google, and the other labs are their own company. The dummy test labs send nothing anywhere and have no provider. The defaults live in LAB_DEFAULT_PROVIDERS in middleman/src/middleman/model_info.py. A recorded value always wins, so only record one when the lab's default is wrong for that model, e.g. hawk proxy models update <name> --config '{"provider": "<company>"}'; a wrong value makes Hawk treat the model as served by that company. Once set, the value can be changed but not reset to null.
It appears in hawk models -v output, except for models whose details are secret, where it's hidden like lab (a pre-release model is usually served by the lab that made it); the admin API still shows it. A model with no provider (one on a dummy lab) reports null, meaning unknown: consumers comparing providers should fail closed rather than assume an unknown provider matches, or differs from, another. Middleman's /model_groups endpoint returns a providers map alongside labs for the queried models, including models whose details are secret. Models with no provider are left out of that map, so treat a missing entry as unknown. The admin API shows only what was recorded, not the lab default.
Per-model Anthropic account routing¶
A model config may carry an anthropic_account field naming one of the profiles configured via hawk:middlemanAnthropicProfiles (or hawk:middlemanAnthropicProfilesJson). Requests for that model then authenticate to Anthropic with that profile's workload-identity-federation (OIDC client-credentials) setup instead of the shared ANTHROPIC_API_KEY; unknown account names are rejected when the model is created or updated. Most deployments don't need this — setting ANTHROPIC_API_KEY covers the common case.
To run without static Anthropic keys, set hawk:middlemanAnthropicDefaultProfile to one of those profiles. Every Anthropic model without its own anthropic_account (including models found by discovery) then uses it on the passthrough API, and model discovery authenticates with it too. Predeployment models (anthropic-chat-predeployment) never use the default: without an account of their own they keep the predeployment keys. A model routed to a profile never falls back to ANTHROPIC_API_KEY when that profile fails. Give the default profile a high_priority_profile (below) to keep high-priority requests on separate capacity. Priority quota groups keep matching these models with baseline_profile: null, and a group's common_profile can't be the default profile.
Requiring anthropic-beta flags¶
A model on an Anthropic lab (anthropic, anthropic-chat or anthropic-chat-predeployment) may carry required_anthropic_betas, a list of flags every request to it must send in the anthropic-beta header. This suits a route that exists to use a beta feature — for example, two model configs pointing at the same Anthropic model through different anthropic_account orgs, where only one org has the feature enabled.
- Middleman validates the header it forwards and never adds a flag to it. A request to
/anthropic/v1/messagesmissing any required flag gets a 400 naming the missing flags; flags are matched exactly (case-sensitive), in any order, alongside any others. If the header is sent more than once, the copies are combined into one comma-separated value, as Anthropic reads them, so the flags may appear in any copy. /anthropic/v1/messages/count_tokensis exempt.- A malformed value in a hand-edited config row (or one on a non-Anthropic lab) fails closed: Middleman logs an error and refuses every request to that model, on every route, with a 500 until it's fixed. The admin API still reads such a row and reports the fault in
required_anthropic_betas_error, so it can be found and cleared. - The list isn't included in public model info (
hawk models), since flag names can reveal unreleased features.
Set it with hawk proxy models update <name> --config '{"required_anthropic_betas": ["example-beta-2025-01-01"]}' and clear it by setting it to null. Entries are trimmed and de-duplicated; an empty list, a blank entry, or one containing a comma, whitespace, a control character or anything other than visible ASCII is rejected, as is setting the field on (or moving a model with it to) a non-Anthropic lab.
Per-model OpenAI account routing¶
The OpenAI counterpart: a model config may carry an openai_account field naming a profile from hawk:middlemanOpenaiProfiles (or hawk:middlemanOpenaiProfilesJson). OpenAI WIF profiles authenticate as the middleman ECS task role (assertion_source: "aws_sts"), so they require hawk:middlemanStsWebIdentityEnabled: "true"; the resulting task-role policy allows both the Anthropic and OpenAI token audiences. When creating the identity provider in the OpenAI dashboard, set its Audience to exactly https://api.openai.com/v1: OpenAI lets you choose this value, but middleman always requests that audience from STS and the task-role policy allows only it. Setting OPENAI_API_KEY remains the common case.
To run without static OpenAI keys, set hawk:middlemanOpenaiDefaultProfile to one of those profiles. Models on the labs that use OPENAI_API_KEY (openai-chat, openai-responses, and the legacy-completions labs) with no openai_account of their own then use it on the passthrough API, and so do batch file uploads and model discovery. The *-with-data-retention labs never use the default: they keep OPENAI_API_KEY_WITH_DATA_RETENTION, which belongs to a different organization. OpenAI priority overflow routes still use the static keys, and saving a priority config with an OpenAI group requires them, so keep OPENAI_API_KEY until those move too. Give the default profile a high_priority_profile to keep a separate high-priority lane.
OpenAI Decisions¶
POST /openai/v1/decisions uses the existing OpenAI passthrough and model aliases. To track Decisions usage separately, add another alias for the same upstream model. For a model using the normal OpenAI credentials:
hawk proxy models add gpt-6-luna-decisions --group model-access-public \
--config '{"lab":"openai-responses","danger_name":"gpt-6-luna","are_details_secret":false}'
With an OpenAI Python client configured for Middleman (SDK 3.26.0 or later):
decision = client.decisions.create(
model="gpt-6-luna-decisions",
input="The package arrived with a dent.",
questions=[{"type": "predicate", "instructions": "Is the package damaged?"}],
)
See OpenAI's Decisions guide for API details.
HIGH-priority profile routing¶
Anthropic and OpenAI account profiles can set high_priority_profile to another profile name in the same provider's configuration. Requests with X-Middleman-Priority: high use that profile's credentials; other requests keep using the model's configured profile. The target can use either an API key or a WIF authentication method supported by that provider. For Anthropic WIF, configure the target profile with the intended HIGH workspace's workspace_id.
For example, this Pulumi configuration routes HIGH requests for models using anthropic_account: regular to the high profile:
hawk:middlemanAnthropicProfiles:
regular:
auth_type: api_key
api_key_env: ANTHROPIC_API_KEY
high_priority_profile: high
high:
auth_type: api_key
api_key_env: ANTHROPIC_HIGH_PRIORITY_API_KEY
Selection happens once: if A points to B and B points to C, a HIGH request starting at A uses B, while one starting at B uses C. The selected target's own high_priority_profile and api_key_env_high are not applied again. Self-references and cycles therefore do not recurse. Referenced names must exist when profiles are loaded; a missing target is a configuration error.
Without high_priority_profile, existing behavior is preserved: API-key profiles use api_key_env_high for HIGH requests when configured, and WIF profiles use their normal credentials. An explicit named HIGH profile takes precedence over the source's api_key_env_high. WIF tokens, refresh locks, and cooldowns belong to the selected profile, so multiple sources selecting the same target share its cached token.
Model group naming
Model groups use the prefix model-access-<name> (e.g. model-access-public), and the user's JWT must carry a matching group. For Cognito users without explicit group claims, hawk:defaultPermissions provides the fallback (default "model-access-public"). The admin group itself must not use the model-access- prefix and must not overlap with defaultPermissions — Middleman refuses to start if either constraint is violated.
Admin gate covers provider keys too
The admin gate covers both model management (/admin/models/) and provider-key rotation (/admin/secrets/provider-keys). Anyone who can run cognito-idp:AdminAddUserToGroup on your user pool can grant admin transitively — treat that IAM permission as equivalent to Middleman admin in policy reviews.
Revoking admin: removing a user from the admin Cognito group does not immediately revoke their existing access token; it stays admin until its TTL expires (1 hour by default for Cognito access tokens). Refresh tokens (30 days default) mint fresh access tokens without the removed claim, so the practical revocation window is the access-token TTL. For immediate revocation, run aws cognito-idp admin-user-global-sign-out --user-pool-id <pool> --username <email> — this invalidates all of that user's tokens.
Usage attribution by user¶
The user dimension on Middleman's token-usage metrics, and the user_email field of its traffic log, carry the caller's email address: Middleman reads the claim named by hawk:oidcEmailField (default email), the same setting the Hawk API uses to identify the person. A token without that claim — a machine client's, or an IdP that never emits one — is attributed to its JWT sub instead, so nothing is dropped; the traffic log always records user_id (the sub) as well, and that is the field to join on. If your IdP puts the address in a different claim, point hawk:oidcEmailField at it and the API, relay and Middleman all follow. On a Hawk-provisioned Cognito stack the access token carries no email claim, so callers stay attributed by sub; getting an email there needs a pre-token-generation Lambda that adds the claim. The dimension is recorded at request time, so per-user series keyed on an opaque subject end at the deploy that enabled this and new ones begin under the email.
Usage attribution by team¶
Middleman emits its token-usage CloudWatch metrics with a team dimension alongside the per-user one, and the viewer's usage page can group by it (/usage?group_by=team). The team comes from the user's access token: Middleman reads the claim named by hawk:oidcTeamsClaim (default teams), strips hawk:oidcTeamsGroupPrefix (default team-) from each value, and uses the result as the team name. A token whose claim is missing or empty is attributed to unassigned; a user in several teams is attributed to the sorted, +-joined combination (e.g. alignment+infra) rather than double-counted. A group whose stripped name is unassigned or contains + would collide with those labels, so Middleman drops it and logs auth.invalid_team_name at error level; rename the group.
The convention is to name IdP groups team-<name> and have the IdP emit them in a teams claim on the access token:
- Okta — on the authorization server, add a Groups claim named
teamsto the access token with filter Starts withteam-. - Other IdPs — any provider works as long as the access token carries a string-list claim of group names; point
hawk:oidcTeamsClaimat it and sethawk:oidcTeamsGroupPrefixto whatever prefix your groups use (or""to keep the full group name). - Set
hawk:oidcTeamsClaim: ""to disable team attribution entirely; every request is then attributed tounassigned.
The dimension is recorded at request time, so team history starts from the deploy that enabled it — there is no backfill of earlier usage, and moving a user between groups only affects requests made after their next token refresh.
Usage attribution by job and task¶
The usage API can also group by job (eval set or scan id, GET /usage?group_by=job) and by Inspect task name (group_by=task). Those groupings are far too high-cardinality for a CloudWatch metric dimension, so the Hawk API answers them from Middleman's traffic log instead: Logs Insights queries over the <env>/middleman/traffic log group, reading each record's public_name, token counts and the x-hawk-job-id / x-hawk-job-type / x-inspect-task-name correlation headers the runner stamps on model requests.
This needs the traffic log at level summary or full (hawk:middlemanTrafficLogLevel; summary is the default). When it is on, the stack sets HAWK_API_MIDDLEMAN_TRAFFIC_LOG_GROUP on the API and grants its task role logs:StartQuery on that log group; at level off the job and task groupings return no data. Token counts are present in the traffic log only for requests since 2026-09-18, and history is bounded by the log group's retention. See Token Usage for the endpoints, label semantics (direct, scan, unknown, other) and caveats.
Deploying Changes¶
Middleman runs on ECS Fargate. Deployments are triggered by pushing to the main branch, which builds a new Docker image and updates the ECS service via CI/CD.
Running Locally¶
Testing the Passthrough API¶
This script tests the passthrough API against multiple providers (Anthropic, OpenAI, OpenRouter).