Skip to content

Self-hosted inference

Hawk can serve open-weights models on its own GPUs: EC2 Capacity Block nodes running vLLM, in any region, behind one HTTPS endpoint that Middleman calls like any other provider. It is off unless the stack sets hawk:inference.

Middleman ─ HTTPS ─> internal ALB ─> vllm-router (ECS, hawk's VPC) ─┬─ IPv6 peering ─> cell us-east-1: GPU nodes
                                                                    └─ IPv6 peering ─> cell …: GPU nodes

How it works

  • One endpoint. Middleman calls https://<hostnameLabel>.<privateDomain>/v1 and knows nothing about cells or nodes. Only Middleman's security group reaches the ALB, which forwards /v1/* only. The routers decide where each request goes (by default cache_aware, which keeps a conversation on the node that holds its prefix cache), across every node of every cell. Current router builds key each request by its prompt tokens, which they get by having a node render it (POST /v1/chat/completions/render and its /v1/completions and /v1/responses siblings: the chat template and tokenizer applied, nothing generated). Releases up to v0.1.16.1 key chat on session_params.session_id, which Middleman doesn't send, and so balance it by load alone.
  • Cells. Each GPU region gets a VPC of its own, peered with hawk's. Its GPU subnet is IPv6-only and its route table holds the local route, the S3 gateway endpoint (weights) and the peering routes back to the router tier, nothing else: no internet, no NAT. Security groups name the router tier's IPv6 ranges, since they can't reference groups across regions. EFA-only interfaces on every network card carry inter-node traffic in a group that admits only itself.
  • No keys. The lab uses credential_kind: none: Middleman sends no credential, and its key-host allowlist admits the endpoint under NO_CREDENTIAL (added automatically). The network is the identity.
  • No way in. Nodes have no SSH key, no SSM agent (Image Builder removes it after the build), and an instance role that can only read the cell's bucket and write its own status there. Nothing on a node changes except through a deploy.
  • Nodes. The engine runs as a rootless container under rootless containerd, with the GPUs, EFA devices and host network. Everything a node runs is baked into its AMI by EC2 Image Builder (built in hawk's region, which has NAT, and copied to the cell's), so nodes never pull images or reach the internet. A node's root volume restores at a provisioned rate rather than on first read, and weights sync from the cell's bucket to instance-store NVMe at boot, at the node's network bandwidth.
  • Engine updates. A change to a cell's engine arguments or model reaches the running nodes in place, in minutes. The deploy writes the engine configuration to the cell's bucket, which starts the cell's rollout (a Step Functions state machine). It tags one node at a time with the new version, which the node sees through instance metadata within a second; the node copies any new weights while its engine still serves, restarts the engine, and reports back through the bucket. A node whose engine doesn't serve goes back to the last configuration that did, and the rollout then takes every node back to the cell's current configuration and fails. Nodes launched later start on the last configuration a rollout completed.
  • Node updates. A change to the node image (including the engine image) builds a new AMI and a new launch template version, and the Auto Scaling group's Rolling instance refresh replaces the nodes one at a time. The block has no spare slot, so each node is terminated before its replacement launches, and the replacement syncs its weights from S3 again. It also gets a new address, which the routers' worker-sync sidecar picks up.
  • Capacity Blocks. A cell always asks for nodes of the block's slots (all of them by default), whether or not the block is active. Before the block starts, after it ends, or while something else that launches into the same reservation holds a slot, the group's launches fail and it keeps retrying. If the group runs fewer nodes than it wants for 30 minutes, a Datadog monitor notifies the stack's notification target, and a person decides what to do. The monitor reads the group's AWS/AutoScaling metrics, so Datadog's AWS integration must collect that namespace (add it to hawk:datadogAwsIntegrationMetrics if that list is set, or wherever the integration is managed). EC2 itself starts terminating the nodes 30 minutes before the block ends. On a clean shutdown the engine gets SIGTERM and drains for up to 60 seconds (--shutdown-timeout), so only requests still running after that are cut. After about a day of failed launches, Auto Scaling may suspend the group's launches. If that happens, resume them (aws autoscaling resume-processes --auto-scaling-group-name <group>), and deploy a new block's cell close to the block's start.
  • Router deploys. A deploy that changes the routers (a new router image, say) starts new ones, then drains each old one for up to an hour, the ALB's limit. A stream still running after that is cut.
  • Worker list. A worker-sync sidecar in each router task asks EC2 for every cell's running nodes every 15 seconds. It adds a node to its router once the engine passes /health and removes nodes that are no longer running, so new and replaced nodes join without a deploy. Until the first node is ready, the router's only worker is a placeholder that answers inference requests with 503.
  • Telemetry. Each node's OpenTelemetry Collector sends the engine's metrics and logs to the routers' Datadog agents through the ALB, over the peering.

Setting it up

  1. Buy the Capacity Block, and decide how many of its slots the cell runs (nodes; two or more so updates keep serving).
  2. Set hawk:inference (see Pulumi.example.yaml) and deploy. The first deploy creates each cell's weights bucket and builds the cell's AMI, which takes most of an hour.

  3. Mirror the weights into the cell's bucket and set the prefix it prints as the cell's engine.modelS3Prefix:

    hawk inference mirror-weights <org>/<model> --revision <branch-tag-or-commit> --cell <cell> --env <env>

    It resolves the revision to a commit and starts the stack's weights mirror, a Fargate task that streams every file from Hugging Face into models/<org>/<model>/<commit>/, many at once, checking each file's hash before it lands. Rerunning it skips the files already there. It needs your AWS credentials, with ssm:GetParameter on /<env>/inference/weights-mirror, ecs:RunTask on the mirror's task definition and iam:PassRole on its two roles (<env>-inference-weights-mirror-exec and -task), and, to follow the task, ecs:DescribeTasks and logs:GetLogEvents on /ecs/<env>-inference-weights-mirror. It fetches only public, ungated models. Weights from elsewhere can be uploaded under any prefix of the bucket instead. Nodes wait for their weights at boot; the routers add each node once its engine serves. 4. Add the labs and models to Middleman. A lab serves one API, so add a lab and a model for each: the anthropic one is what Claude Code calls (its base_url has no /v1, since Middleman appends /v1/messages).

    hawk proxy labs add self-hosted --config '{"api": "openai", "base_url": "https://inference.<privateDomain>/v1",
  "credential_kind": "none", "provider": "self-hosted"}'
hawk proxy labs add self-hosted-anthropic --config '{"api": "anthropic",
  "base_url": "https://inference.<privateDomain>", "credential_kind": "none", "provider": "self-hosted"}'
hawk proxy models add <public-name> --group <group> --config '{"lab": "self-hosted", "danger_name": "<servedModelName>",
  "are_details_secret": true, "is_zdr": true, "raw_cot": true}'
hawk proxy models add <public-name>-anthropic --group <group> --config '{"lab": "self-hosted-anthropic",
  "danger_name": "<servedModelName>", "are_details_secret": true, "is_zdr": true, "raw_cot": true}'

    raw_cot is true for a model whose responses carry its unsummarized reasoning, as an open-weight reasoning model's do through vLLM, and false for one that returns none.

    The labs' provider: self-hosted marks their models as served on your own infrastructure: Hawk's /traffic API never serves those models' request and response bodies, since they may be sent other models' raw chain-of-thought. That is decided from the model's current lab, so don't later re-point one of these model names at an external lab, or change these labs' provider; register a new name (or lab) for the external route instead (security).