Skip to content

Self-hosted inference

Hawk can serve open-weights models on its own GPUs: GPU nodes running vLLM, in any region, behind one HTTPS endpoint that Middleman calls like any other provider. A cell of nodes runs either on an EC2 Capacity Block, or as plain on-demand instances (such as g6 or g6e, which have no Capacity Blocks of their own to buy). It is off unless the stack sets hawk:inference.

Middleman ─ HTTPS ─> internal ALB ─> vllm-router (ECS, hawk's VPC) ─┬─ IPv6 peering ─> cell us-east-1: GPU nodes
                                                                    └─ IPv6 peering ─> cell …: GPU nodes

How it works

  • One endpoint. Middleman calls https://<hostnameLabel>.<privateDomain>/v1 and knows nothing about cells or nodes. Only Middleman's security group reaches the ALB, which forwards /v1/* only. The routers decide where each request goes (by default cache_aware, which keeps a conversation on the node that holds its prefix cache), across every node of every cell. Current router builds key each request by its prompt tokens, which they get by having a node render it (POST /v1/chat/completions/render and its /v1/completions and /v1/responses siblings: the chat template and tokenizer applied, nothing generated). Releases up to v0.1.16.1 key chat on session_params.session_id, which Middleman doesn't send, and so balance it by load alone.
  • Cells. Each GPU region gets a VPC of its own, peered with hawk's. Its GPU subnet is IPv6-only and its route table holds the local route, the S3 gateway endpoint (weights) and the peering routes back to the router tier, nothing else: no internet, no NAT. Security groups name the router tier's IPv6 ranges, since they can't reference groups across regions. Where the instance type carries EFA — the P family, and the g6 and g6e sizes from 8xlarge up — EFA-only interfaces carry inter-node traffic in a group that admits only itself; the smaller g6/g6e sizes have no EFA devices, so their nodes attach just the primary interface.
  • No keys. The lab uses credential_kind: none: Middleman sends no credential, and its key-host allowlist admits the endpoint under NO_CREDENTIAL (added automatically). The network is the identity.
  • No way in. Nodes have no SSH key, no SSM agent (Image Builder removes it after the build), and an instance role that can only read the cell's bucket and write its own status there. Nothing on a node changes except through a deploy.
  • Nodes. The engine runs as a rootless container under rootless containerd, with the GPUs, EFA devices and host network. Everything a node runs is baked into its AMI by EC2 Image Builder (built in hawk's region, which has NAT, and copied to the cell's), so nodes never pull images or reach the internet. A node's root volume restores at a provisioned rate rather than on first read, and weights sync from the cell's bucket to instance-store NVMe at boot, at the node's network bandwidth.
  • Engine updates. A change to a cell's engine arguments or model reaches the running nodes in place, in minutes. The deploy writes the engine configuration to the cell's bucket, which starts the cell's rollout (a Step Functions state machine). It tags one node at a time with the new version, which the node sees through instance metadata within a second; the node copies any new weights while its engine still serves, restarts the engine, and reports back through the bucket. A node whose engine doesn't serve goes back to the last configuration that did, and the rollout then takes every node back to the cell's current configuration and fails. Nodes launched later start on the last configuration a rollout completed.
  • Node updates. A change to the node image (including the engine image) builds a new AMI and a new launch template version, and the Auto Scaling group's Rolling instance refresh replaces the nodes one at a time. A block cell has a slot per node and no spare, so each node is terminated before its replacement launches; an on-demand group keeps one spare slot, so its replacement launches (and syncs its weights from S3 again) first. It also gets a new address, which the routers' worker-sync sidecar picks up. Each replacement batch holds the refresh for one warm-up before the next: a block cell waits out its start-up grace (70 minutes by default) and replaces as many nodes at a time as leave half of them serving (a lone node: all of it), while an on-demand cell replaces exactly one node per warm-up — and waits out both of a node's budgets, its grace to copy weights and the same again to start its engine, since only then has its predecessor stopped serving. So a block refresh waits on its half-batches and an on-demand refresh on every node; healthCheckGraceSeconds sizes each wait. A deploy that ships a new node or engine image changes the launch template, which a running refresh will not accept: it fails with a validation error rather than waiting its turn. Let the refresh finish, or cancel it (aws autoscaling cancel-instance-refresh) and let the new deploy start a fresh one. Until the refresh-failure alert ships (PLT-1341), confirm a node-image deploy by hand: aws autoscaling describe-instance-refreshes --auto-scaling-group-name <env>-inference-<cell> should show the latest refresh in Successful.
  • Capacity blocks. A cell always asks for nodes of the block's slots (all of them by default), whether or not the block is active. Before the block starts, after it ends, or while something else that launches into the same reservation holds a slot, the group's launches fail and it keeps retrying. If the group runs fewer nodes than it wants for 30 minutes, a Datadog monitor notifies the stack's notification target, and a person decides what to do. The monitor reads the group's AWS/AutoScaling metrics, so Datadog's AWS integration must collect that namespace (add it to hawk:datadogAwsIntegrationMetrics if that list is set, or wherever the integration is managed). EC2 itself starts terminating the nodes 30 minutes before the block ends. On a clean shutdown the engine gets SIGTERM and drains for up to 60 seconds (--shutdown-timeout), so only requests still running after that are cut. After about a day of failed launches, Auto Scaling may suspend the group's launches. If that happens, resume them (aws autoscaling resume-processes --auto-scaling-group-name <group>), and deploy a new block's cell close to the block's start.
  • On demand. A cell without a block (onDemand) launches plain on-demand instances — the only way to run the g6 and g6e families, which have no Capacity Blocks of their own to buy — and nodes says how many (it has no block size to default from). The group's floor is that count, so nothing takes it below — and nodes: 0 pauses the cell: its instances terminate, while the cell's bucket, VPC and AMI stay, since removing the cell from the config would destroy those too. The same 30-minute monitor watches these groups, where a short on-demand vCPU quota or scarce EC2 capacity looks just like a block that hasn't started: launches that keep failing. A cell's zone is fixed at creation — its GPU subnet carries the cell VPC's one IPv6 block, so editing availabilityZone in place just fails the deploy when the replacement subnet can't take the CIDR the old one still holds — so moving zones means removing the cell and adding a new one in the new zone.
  • Router deploys. A deploy that changes the routers (a new router image, say) starts new ones, then drains each old one for up to an hour, the ALB's limit. A stream still running after that is cut.
  • Worker list. A worker-sync sidecar in each router task asks EC2 for every cell's running nodes every 15 seconds. It adds a node to its router once the engine passes /health and removes nodes that are no longer running, so new and replaced nodes join without a deploy. Until the first node is ready, the router's only worker is a placeholder that answers inference requests with 503.
  • Telemetry. Each node's OpenTelemetry Collector sends the engine's metrics and logs to the routers' Datadog agents through the ALB, over the peering.

Setting it up

  1. Buy the Capacity Block, and decide how many of its slots the cell runs (nodes; two or more so updates keep serving) — or skip it and configure the cell as onDemand with an instanceType (any g6 or g6e size works out of the box; pick for the model, not the node: an L4 brings 24 GB of GPU memory per GPU and an L40S 48, and the weights share that memory with vLLM's token cache and working set, so leave room beyond them — and keep parallelism inside the node, --tensor-parallel-size × dataParallelSize at most the node's GPU count), an availabilityZone and a nodes count. Raise the Running On-Demand G and VT vCPU quota to nodes + 1 nodes' worth of the type above whatever the region already draws from it — the stack's own default Karpenter GPU pool (g4dn) launches on-demand G instances against the same quota — and do it before the first deploy (a 48xlarge takes 192 vCPUs), because the on-demand cell's refreshes launch each replacement into a spare slot while the node it replaces still runs, Capacity Blocks never touch the quota, and a launch refused on quota shows up only as nodes that never appear while the group retries.
  2. Set hawk:inference (see Pulumi.example.yaml) and deploy. The first deploy creates each cell's weights bucket and builds the cell's AMI, which takes most of an hour.
  3. Mirror the weights into the cell's bucket and set the prefix it prints as the cell's engine.modelS3Prefix:

    hawk inference mirror-weights <org>/<model> --revision <branch-tag-or-commit> --cell <cell> --env <env>
    

    It resolves the revision to a commit and starts the stack's weights mirror, a Fargate task that streams every file from Hugging Face into models/<org>/<model>/<commit>/, many at once, checking each file's hash before it lands. Rerunning it skips the files already there. It needs your AWS credentials, with ssm:GetParameter on /<env>/inference/weights-mirror, ecs:RunTask on the mirror's task definition and iam:PassRole on its two roles (<env>-inference-weights-mirror-exec and -task), and, to follow the task, ecs:DescribeTasks and logs:GetLogEvents on /ecs/<env>-inference-weights-mirror. It fetches only public, ungated models. Weights from elsewhere can be uploaded under any prefix of the bucket instead. Nodes wait for their weights at boot; the routers add each node once its engine serves. 4. Add the labs and models to Middleman. A lab serves one API, so add a lab and a model for each: the anthropic one is what Claude Code calls (its base_url has no /v1, since Middleman appends /v1/messages).

    hawk proxy labs add self-hosted --config '{"api": "openai", "base_url": "https://inference.<privateDomain>/v1",
      "credential_kind": "none", "provider": "self-hosted"}'
    hawk proxy labs add self-hosted-anthropic --config '{"api": "anthropic",
      "base_url": "https://inference.<privateDomain>", "credential_kind": "none", "provider": "self-hosted"}'
    hawk proxy models add <public-name> --group <group> --config '{"lab": "self-hosted", "danger_name": "<servedModelName>",
      "are_details_secret": true, "is_zdr": true, "raw_cot": true}'
    hawk proxy models add <public-name>-anthropic --group <group> --config '{"lab": "self-hosted-anthropic",
      "danger_name": "<servedModelName>", "are_details_secret": true, "is_zdr": true, "raw_cot": true}'
    

    raw_cot is true for a model whose responses carry its unsummarized reasoning, as an open-weight reasoning model's do through vLLM, and false for one that returns none.

    The labs' provider: self-hosted marks their models as served on your own infrastructure: Hawk's /traffic API never serves those models' request and response bodies, since they may be sent other models' raw chain-of-thought. That is decided from the model's current lab, so don't later re-point one of these model names at an external lab, or change these labs' provider; register a new name (or lab) for the external route instead (security).