New Environment Setup¶
Prerequisites¶
- AWS account with appropriate permissions
- Pulumi CLI installed
- AWS CLI configured
- A domain name for your services
- Sufficient Fargate/EC2 vCPU quotas — see sizing and quotas; request increases on day one, brand-new accounts are often denied
Quick Start¶
1. Install prerequisites¶
2. Authenticate¶
Default region
Set the default region to the region you'll deploy to (aws:region in your stack config, e.g. us-west-2) — the setup commands below create resources in that region. With an SSO profile, the profile's region applies; override with export AWS_DEFAULT_REGION=us-west-2 if it differs.
3. Set up Pulumi state backend¶
Create an S3 bucket and KMS key:
# Suffixing your account ID makes the bucket name globally unique
BUCKET="hawk-pulumi-state-$(aws sts get-caller-identity --query Account --output text)"
aws s3 mb "s3://$BUCKET" --region <region>
aws kms create-alias --alias-name alias/pulumi-secrets --region <region> \
--target-key-id $(aws kms create-key --region <region> --query KeyMetadata.KeyId --output text)
pulumi login "s3://$BUCKET?region=<region>&awssdk=v2"
pulumi whoami -v # Backend URL must show YOUR bucket before any stack operation
Credential troubleshooting
If pulumi login fails with NoCredentialProviders or AccessDenied: No AWSAccessKey was presented, your AWS credentials aren't visible to Pulumi. Make sure you ran aws configure (not just aws login). If using SSO profiles or short-lived credentials, ensure AWS_PROFILE is set, or export credentials explicitly. A failed login leaves Pulumi on the previous backend — verify pulumi whoami -v before any stack operation:
Don't use aws login because its credentials are short-lived (e.g. 12h), so they can expire mid-deploy or before you tear down. Use aws configure for deploys.
4. Choose a domain and DNS strategy¶
Hawk's services run under hawk.<hawk:domain> — for example, with hawk:domain: example.com the API is at api.hawk.example.com. Pulumi adds the hawk. segment automatically; do not include it in the config value. Each service is protected by an ACM TLS certificate. Pulumi needs a Route 53 public hosted zone for hawk:publicDomain and DNS delegation working before certs can validate. Pick one path before configuring the stack:
- (a) Route 53 Domains — AWS handles registration, zone creation, and delegation. Leave
hawk:createPublicZone: "false"(the default). -
(b) Existing registrar + manual delegation — Create or import a Route 53 public hosted zone first, paste its four NS records into your registrar, and leave
hawk:createPublicZone: "false"so preflight can validate delegation before deploy.# The zone is named after your hawk:publicDomain value (e.g. example.com) — # NOT hawk:domain; services live on hawk:domain subdomains INSIDE this zone ZID=$(aws route53 create-hosted-zone --name <your hawk:publicDomain> \ --caller-reference "$(date +%s)" --query 'HostedZone.Id' --output text) aws route53 get-hosted-zone --id "$ZID" \ --query 'DelegationSet.NameServers' --output text # add these 4 NS records at your registrar -
(c) Cloudflare delegation — Parent zone in Cloudflare; create or import the Route 53 public hosted zone first, then delegate the subdomain from Cloudflare before deploy. See the Configuration Reference for related setup.
- (d) Skip TLS — testing only — Set
hawk:skipTlsCerts: "true"for an HTTP-only deploy. Browser approval review is unavailable in this mode: its remote relay connection requires HTTPS/WSS to protect the reviewer's token. Enable TLS to use browser approvals.
See the Configuration Reference: Domain & DNS for full details. Note: if DNS isn't working when pulumi up runs, the wildcard ACM cert validation will hang for ~75 min before failing — get delegation in place first and run preflight before deploy.
5. Create and deploy¶
Run from the repo root:
# Copy the example config FIRST — `pulumi stack init` merges its KMS metadata
# (secretsprovider/encryptedkey) into the existing file; copying afterwards
# would overwrite that metadata and break secret encryption.
cp Pulumi.example.yaml Pulumi.my-org.yaml
pulumi stack init my-org \
--secrets-provider="awskms://alias/pulumi-secrets?region=<same as aws:region>&awssdk=v2"
# Edit Pulumi.my-org.yaml with your values after completing DNS delegation from step 4
docker login # required — Docker Hub (https://hub.docker.com/) rate-limits anonymous pulls
docker login dhi.io # required — Hawk's Python base images live on dhi.io (free Community tier; same Docker Hub credentials)
scripts/dev/preflight.sh
pulumi up
First deploy creates ~400 AWS resources and typically takes 30-45 minutes — building and pushing the container images dominates on a cold Docker cache.
6. Secrets¶
For production stacks, set up API keys. This goes through Middleman's admin API, so it needs the hawk CLI, a hawk login session, and Middleman admin:
For dev environments, secrets are automatically shared from staging — no manual seeding needed.
To enable runners to clone private GitHub repos, configure the git credentials secret:
The token is stored in Secrets Manager and readable by anyone with access. Use a fine-grained PAT with minimal scope.
To keep it out of runners instead, create a GitHub App with only the Contents: read-only repository permission, no webhook, and installable on any account. Put its private key in <env>/hawk/github-app-key, then set its ID and deploy:
aws secretsmanager put-secret-value --secret-id <env>/hawk/github-app-key --secret-string file://app.private-key.pem
pulumi config set hawk:githubAppId 123456
Each organization whose private repositories runs use installs the App itself. Hawk asks GitHub which installation covers a repository, so adding one needs no change or restart. Use a separate App for each environment, with staging's installed only on a few test repositories, so a key leaked from staging can't read production's repositories.
The secret's policy lets only the token broker's role read the key, plus any break-glass principals you list in hawk:githubAppKeyBreakGlassPrincipals. Dev environments use staging's App, as they use its git-config secret, so staging's policy also lets their token brokers (dev-*-inspect-ai-token_broker-lambda) read it; deploy staging first. Anyone who can deploy a dev stack can therefore read staging's key, whatever the list says, which is why staging's App belongs on test repositories only.
Runners then get a one-hour, single-repository token per private repository from the token broker (see runner Git access). Install the App on Only select repositories: every user who can launch a run can read every private repository the App is installed on that no code-access rule restricts. All repositories also covers repositories created later, but exposes every private repository in the organization that way.
Once hawk:githubAppId is set, runners receive nothing from <env>/inspect/api-git-config, its other entries included, and the API refuses to start without the token broker rather than keep handing runners that credential. The broker re-reads the key every five minutes, so writing a new one needs no restart. Unsetting the App ID later leaves runners launched with it without Git credentials for their remaining fetches: drain them first unless it's an emergency.
7. Your services¶
After deployment, your services are available at:
- API:
https://api.hawk.<domain> - Viewer:
https://viewer.hawk.<domain> - Middleman:
https://middleman.<domain>(ormiddleman.<privateDomain>when a private domain is configured)
8. View API logs¶
aws logs tail "$(pulumi stack output api_log_group_name)" \
--region <region> --since 30m --format short | grep -v /health
Dev Environments¶
For development, create lightweight environments that share staging infrastructure:
Requires PULUMI_BACKEND_URL to be exported and a deployed stack literally named stg in that backend (the script clones its config — if your shared/staging stack has a different name, the script exits with an error; configure the dev stack manually instead). The script configures the stack and prompts to deploy. Your dev environment shares staging's VPC, ALB, and EKS cluster while getting its own database and services.
See Deployment for more details on managing dev environments.
Tailscale VPN Setup¶
If using Tailscale for private service access:
- Pick an IPv4 CIDR that doesn't conflict with existing ranges (e.g.,
10.13.0.0/16) - Add the CIDR to your Tailscale ACL rules
- Add a custom nameserver for
<environment>.<your-domain>pointing to the CIDR's DNS resolver (e.g.,10.13.0.2) - Set
hawk:albInternal: "true"in your stack config - Store a Tailscale auth key in AWS Secrets Manager
- Deploy with
pulumi up