Priority and per-user allocation¶
Middleman accepts x-middleman-priority: low | medium | high. Header values ignore case and surrounding whitespace. Requests without this header default to MEDIUM; unrecognized header values fall back to MEDIUM. Explicit LOW, MEDIUM, and HIGH values are preserved. Hawk's priority setting sets this header, with evaluations defaulting to MEDIUM and scans to LOW. Fairness uses the authenticated user's identity. Callers are trusted to choose their priority; sustained higher-priority demand can leave lower classes with no allocation.
The policy is opt-in for explicitly mapped Anthropic and OpenAI quota groups. It allocates a verified token or request budget in HIGH, MEDIUM, LOW order, with max-min fairness between users in each class. Demand uses a 20-second arrival window and completed usage supplies historical costs. Admission uses local stride counters; fleet observations are exchanged through Valkey approximately once per second. A single active user at a single priority bypasses local shedding and uses the common route when enforcement is eligible; provider quotas still apply. Multiple active priorities compete in class order even when they belong to the same user. There is no prediction of the next request's size or capacity estimate learned from 429 rates.
Preserve the reservation baseline¶
To preserve the provider reservation, an operator must manually set the ordinary workspace's per-model-family limits in the Anthropic Console to 95% of the shared organization quota. LOW and MEDIUM share this cap; HIGH may use the full parent quota and is not capped at 5%. Middleman selects credentials but does not create or maintain these provider limits. For WIF, each profile has its own workspace_id. Set high_priority_profile on the ordinary profile to the name of a separate profile for HIGH. Both workspaces must belong to the intended organization and be accessible to their profiles' federation identities. Omitting high_priority_profile preserves the single-workspace route.
For example, the regular profile can have workspace_id: ws-ordinary and high_priority_profile: urgent, while the urgent profile has workspace_id: ws-high.
Workspace limits are absolute, model-family-specific values. New models may inherit an uncapped parent and quota increases can leave an old ordinary cap below 95%. Check parent and workspace limits in the provider Console for each model family. Ordinary-workspace response headers do not establish parent quotas.
Configure selected groups¶
Administrators can GET and PUT /admin/priority. PostgreSQL stores the complete configuration in middleman.priority_config; apply the database migration before using this endpoint. Existing cache notifications trigger refresh, with a five-second periodic refresh if a notification is lost. Local development without a database can use MIDDLEMAN_PRIORITY_CONFIG containing the same JSON.
The default configuration is:
Each group supplies name, provider (anthropic or openai), aliases, baseline_profile, common_profile, verified, quota_scope, dimension, budget_per_minute, budget_source, and prior_cost. Anthropic supports requests, input_tokens, and output_tokens. OpenAI supports requests and combined tokens. input_cost distinguishes Anthropic uncached input alone from uncached input plus cache creation. All public aliases sharing one provider model-family quota must map to the same group; unrelated quotas must remain separate.
For Anthropic, budget_per_minute is the initial and fallback parent-quota estimate, not a fixed cap. Matching verified-parent response headers replace it in either direction for up to five minutes after the latest observation, so provider quota increases do not require updating an admission ceiling.
Obtain limits and model-family membership for the selected group from provider administration or existing documented configuration. verified asserts that the quota identity and credential routes have been checked; it does not claim an exact per-request quota-debit formula or proven throughput benefit. Record the budget source and cost assumptions. Production groups remain unverified until their routing facts are checked; a local simulation cannot establish those facts.
Shadow mode can evaluate provisional groups with verified: false and quota_scope: "unknown", using estimated budget_per_minute and prior_cost values. Label budget_source as an estimate; common_profile can remain unset. Proposed decisions preserve actual admission and credential routing. Unverified response headers do not replace these configured estimates. Shadow observations do not verify quota identity or demonstrate enforcement throughput; verification remains required for enforcement.
For Anthropic, baseline_profile must match the model's existing anthropic_account; null preserves static-key routing. common_profile names a configured credential profile that reaches the intended shared quota without the ordinary child cap. Both ordinary and HIGH requests use its default credential in enforcement mode. Preserve baseline profiles and workspace caps for rollback.
For OpenAI, baseline_profile names the ordinary provider key: OPENAI_API_KEY or OPENAI_API_KEY_WITH_DATA_RETENTION. common_profile names the HIGH key in that same retention family and must differ from baseline_profile. These are key-store names, never credential values. Select a key that reaches the intended shared quota; the name alone does not establish its organization, project, or limits. Chat Completions and Responses aliases can share a group when they share the quota. OpenAI-compatible providers and models with an explicit openai_account are excluded; profiled models retain their configured credentials and bypass allocator accounting.
For an OpenAI token budget, historical cost is reported input plus output tokens. Cached input and reasoning tokens already belong to those totals and are not added again. This is an estimate of quota consumption: provider reservation rules and output caps can make actual debit differ. Set a representative initial prior_cost, use the best available budget, and adjust from the first enforcement trial. Missing usage does not become a zero-cost completion. Exact debit calibration is not a launch prerequisite.
OpenAI's effective token budget is the minimum of the configured budget and applicable generic/project token limits observed on the common route. It represents capacity available on that route, not necessarily the organization limit alone. Missing headers retain the last known conservative cap for the configuration's lifetime. Each worker replaces a cap only with a new observation of that same limiter; the fleet takes the conservative minimum. The configured OpenAI budget is an operator ceiling: update it when verified limits rise above it, or reconfigure after an increase to clear older worker observations. Anthropic common-route headers retain their existing ability to update the configured starting budget upward or downward.
| Mode | Admission | Credential route |
|---|---|---|
off |
Existing behavior | Existing ordinary/HIGH route |
shadow |
Existing behavior; log proposed decisions | Existing ordinary/HIGH route |
enforce |
Apply local decisions for eligible groups | Explicit common-quota profile |
| Any fallback | Existing behavior | Existing ordinary/HIGH route |
Enforcement requires an explicitly configured expected_workers count, fresh matching fleet/configuration state, verified quota identity and a common profile. The first trial should use a fixed fleet size. A changed or incomplete registry restores baseline behavior; this does not provide authoritative discovery of arbitrary autoscaling workers. A worker that has never published cannot enforce locally. Allow a 20-second observation warmup after membership/configuration changes. Graceful shutdown deregisters a drained worker before closing its Valkey connection. After a crash or failed deregistration, its entry remains for up to 300 seconds; enforcement then needs the warmup, giving roughly 321 seconds of fallback from the last publication. A staggered rolling replacement can extend the fallback period. The implementation retains bounded histories and evicts idle flows, so its one-second aggregation and idle-history eviction are explicit adaptations of the tested policy.
Unknown identities, stale state, unavailable Valkey, invalid configuration, an unavailable common credential, or an exhausted nonbinding quota dimension restore baseline admission and routing. Only matching verified-parent headers can invalidate a group for nonbinding-dimension exhaustion. Anthropic nonbinding-dimension invalidation remains latched until configuration changes; OpenAI uses a short-lived fleet observation. Exhausting OpenAI generic or project tokens does not disable a token allocator: those are the constraints it controls. Existing in-flight generations finish on the route already selected. Configuration reads and writes are bounded; a worker whose configuration refresh stops also expires its enforcement eligibility.
Observe and validate¶
Traffic summaries record requested priority, authenticated principal, quota group, configured mode, effective route, proposed and actual admission, budget provenance, and fleet coverage. Request, input-token, output-token, combined-token and project-token headers remain separate; headers with unknown scope do not replace the configured budget. Provider refusals and Middleman admission refusals have separate telemetry. A provider 429 without enough evidence retains an unknown capacity classification.
Token-counting metadata calls bypass admission accounting and usage telemetry. Responses compaction also bypasses admission accounting, but contributes to request, usage and quota telemetry. OpenAI background requests preserve baseline routing and bypass accounting; background retrieval/cancellation is outside this adapter's coverage. Do not select a group whose contention is dominated by bypassed traffic and expect full control of that traffic. Chat Completions and Responses strip caller-supplied OpenAI organization/project headers; credentials determine the quota scope.
OpenAI Decisions bypasses priority admission, shared-quota routing, and allocator accounting; normal credentials and usage telemetry still apply.
Local refusals use the provider's retryable error shape and bounded Retry-After, with no override of SDK retry policy. Known usage from partial or failed responses remains in usage telemetry; it must not be counted as useful completed work when evaluating the policy.
OpenAI token-cost samples require successful HTTP delivery, clean EOF, complete numeric input/output usage and a terminal foreground response. Request quotas count each terminal response as one even when token usage is absent. Capped output (length or Responses incomplete) supplies a cost sample because it consumed tokens, but is not a useful-success measurement. Failed or disconnected responses do not train the historical mean. Anthropic retains its complete-Messages accounting contract.
The first enforcement trial is the throughput experiment. Run focused local checks of routing, accounting, client retries, fleet fallback and the off switch. Then configure one selected whole quota group, confirm fleet health through the 20-second warmup, and enable enforcement. Shadow is optional for a brief operational check; a long shadow period and naturally occurring HIGH/LOW traffic are not prerequisites. All-MEDIUM traffic exercises fairness within that class. A small labelled HIGH probe can exercise priority treatment until users start choosing it.
During enforcement, watch per-user completion, retry exhaustion, waiting time, HIGH latency, ordinary-peer progress and total completed work. Include unfinished work; fewer provider 429s alone is not success. The Anthropic benchmark is the maintained ordinary-95% / HIGH-100% reservation. Use the actual existing OpenAI routing as its baseline. Do not split callers sharing a quota between experimental arms.
Set mode to off promptly if useful work falls, callers stall, retry exhaustion worsens, or routing/accounting is wrong. Verify that both effective routing and admission return to baseline on every worker; existing in-flight requests finish on their selected route. Keep baseline provider caps and credentials intact. If service is healthy, leave enforcement enabled and refine it from observed behavior. No quota groups are enabled by default.