Managing Your Deployment¶
Day-to-day Commands¶
pulumi up # deploy changes
pulumi preview # preview without deploying
pulumi stack output --json # view outputs (API URL, DB endpoint, etc.)
pulumi refresh # sync Pulumi state with actual AWS resources
Updating Hawk¶
Pull the latest code and redeploy:
Database migrations run automatically during deployment.
Updating Inspect AI or Scout¶
Use the release preparation script:
scripts/ops/prepare-release.py --inspect-ai 0.3.50 # specific PyPI version
scripts/ops/prepare-release.py --inspect-ai abc123 # specific git commit
scripts/ops/prepare-release.py --inspect-scout 0.2.10 # update Scout
Then deploy and run smoke tests to validate.
Runner Default Environment Variables¶
Pulumi creates <env>/hawk/runner-default-env containing {}. Write key/value pairs into it to inject env vars into every runner job (e.g. shared tokens not routed through Middleman):
aws secretsmanager put-secret-value \
--secret-id <env>/hawk/runner-default-env \
--secret-string '{"WANDB_API_KEY": "..."}'
Runtime values (auth tokens, Sentry, provider secrets) and user-supplied --secret overrides take precedence. Cache TTL: ~5 min.
Smoke Tests¶
Validate that a deployed environment is working end-to-end:
hawk login
scripts/dev/smoke # test current stack
scripts/dev/smoke --stack my-org # test a specific stack
scripts/dev/smoke --warehouse # include database checks
scripts/dev/smoke -k test_real_llm # run a specific test
Smoke tests submit real evals against real models and verify results end up in the warehouse and viewer.
After updating Inspect AI or Scout dependencies:
Tearing Down¶
The easiest path is the teardown script, which automates the whole sequence below (confirmation prompt, deletion-guard removal, bounded Karpenter drain, destroy, stack removal):
Expect a full teardown to take well over an hour — EKS, RDS, NAT, and VPC deletion alone commonly run ~1h15m; that's AWS-side deletion time, not something Hawk can speed up.
To tear down manually, run these two phases in order:
# Phase 1: remove the deletion guards declaratively.
# protectResources=false flips — in one `pulumi up` — the pulumi protect flags,
# the ALB's deletion protection, force_destroy on S3 buckets, and force_delete
# on ECR repos. (`pulumi state unprotect` alone is NOT enough: it clears the
# state flags but leaves those AWS-side guards baked into the resources.)
pulumi config set hawk:protectResources false
pulumi up --yes
# Phase 2: destroy and remove the stack.
PULUMI_K8S_DELETE_UNREACHABLE=true pulumi destroy --yes
pulumi stack rm # remove the stack from Pulumi state
Phase 1 is a regular pulumi up, so it needs the same prerequisites as a deploy (Docker running and logged in for the image builds). If the deployment is too broken for pulumi up to succeed, fall back to pulumi state unprotect --all --yes and expect to handle the ALB/S3/ECR guards manually (see Troubleshooting teardown).
Warning
Always wait for pulumi destroy to finish before running stack rm. Running stack rm first will orphan AWS resources in your account. Don't pipe long-running destroys through tee — it masks Pulumi's non-zero exit code as success.
Troubleshooting teardown¶
Each of these blocks or hangs a pulumi destroy; phase 1 prevents most of them, and the rest have a manual fix.
Unreachable EKS provider¶
If the cluster is gone or the provider's auth in state has gone stale, pulumi destroy fails early with configured Kubernetes cluster is unreachable. PULUMI_K8S_DELETE_UNREACHABLE=true lets Pulumi drop those Kubernetes resources from state and continue.
Karpenter NodeClaims hang the destroy silently¶
Destroy deletes the NodePools, but a NodeClaim's finalizer waits for its node to drain — and a node holding pods that refuse eviction (e.g. runner/sandbox pods carry karpenter.sh/do-not-disrupt) never finishes. The destroy then sits with no error, potentially for 40+ minutes. teardown.sh avoids this by draining NodeClaims with a bounded wait before the destroy; if you hit it manually, force-finalize the stuck NodeClaim:
# find the stuck claim + its EC2 instance
kubectl get nodeclaims
aws ec2 terminate-instances --instance-ids <instance-id>
kubectl patch nodeclaim <name> -p '{"metadata":{"finalizers":null}}' --type=merge
ALB refuses deletion (OperationNotPermitted)¶
The ALB ships with deletion protection while protectResources is on, and it blocks the VPC teardown behind it. Phase 1 turns it off; if you skipped phase 1, disable it out-of-band:
aws elbv2 modify-load-balancer-attributes --load-balancer-arn <alb-arn> \
--attributes Key=deletion_protection.enabled,Value=false
Non-empty S3 buckets and ECR repos¶
BucketNotEmpty: phase 1 sets force_destroy so this shouldn't happen; if you skipped it, empty the bucket first — versioned buckets need all object versions and delete markers removed, not just aws s3 rm --recursive. RepositoryNotEmptyException: likewise covered by phase 1 (force_delete); otherwise delete the images first with aws ecr batch-delete-image.
Redeploying to the same account within 30 days¶
If the stack was destroyed while protectResources was still true, its Secrets Manager secrets were deleted with a 30-day recovery window, and the next pulumi up fails with "secret is already scheduled for deletion" on every <env>/… name. Phase 1 avoids this (with protectResources: false secrets delete immediately); to recover after the fact, force-purge the colliding secrets and re-run the deploy:
aws secretsmanager list-secrets \
--query "SecretList[?starts_with(Name, '<env>/')].Name" --output text | tr '\t' '\n' |
xargs -I{} aws secretsmanager delete-secret --secret-id {} --force-delete-without-recovery
Recovering from an interrupted destroy
An interrupted pulumi destroy (Ctrl-C, crash, dropped connection) can leave a stale lock and pending operations, so pulumi stack rm fails with a lock error or "still has resources". Don't use pulumi stack rm --force to get past it — that removes the stack from state without deleting the resources (the same orphaning the warning above describes).
# Release the stale lock. The lock error names the host + pid holding it;
# confirm that process is gone before cancelling (if the host isn't yours,
# ask its owner) — cancelling a live operation can corrupt state.
# https://www.pulumi.com/docs/iac/cli/commands/pulumi_cancel/
# "already completed" means there was no lock — ignore it.
pulumi cancel -s <stack>
# Pending deletes resolve automatically; pending creates prompt for a choice.
pulumi refresh -s <stack>
pulumi destroy -s <stack>
Cleaning up bootstrap resources¶
pulumi destroy does not remove the resources you created manually before the first deploy — those live outside any stack: the S3 state bucket and KMS key from Quick Start step 3, and, if you pre-created/delegated one, the Route 53 public hosted zone for hawk:publicDomain. The KMS key costs $1.00/month (prorated hourly) until you schedule it for deletion; billing stops as soon as it's scheduled. Run these after pulumi stack rm completes, using the same region as bootstrap:
# Delete the state bucket
aws s3 rb s3://<state-bucket-name> --force
# Schedule the KMS key for deletion (minimum 7 days, default 30)
KEY_ID=$(aws kms describe-key --key-id alias/pulumi-secrets --region <region> \
--query 'KeyMetadata.KeyId' --output text)
aws kms delete-alias --alias-name alias/pulumi-secrets --region <region>
aws kms schedule-key-deletion --key-id "$KEY_ID" --pending-window-in-days 7 --region <region>
# Delete the public hosted zone (must contain only its NS + SOA records;
# delete any other records first)
aws route53 delete-hosted-zone --id <zone-id>
Finally, remove the NS delegation for the zone at your registrar or parent DNS provider (e.g. the four NS records you added in Cloudflare) — otherwise it dangles, pointing at a zone that no longer exists.
Versioned state bucket
If you enabled versioning on the state bucket (Pulumi recommends it for state recovery), aws s3 rb --force will not remove object versions or delete markers and fails with BucketNotEmpty. Empty the bucket first using the S3 console's Empty action or aws s3api delete-objects + aws s3api list-object-versions, then re-run rb.
Troubleshooting¶
Pulumi State Issues¶
# See what Pulumi thinks exists vs what's actually in AWS
pulumi refresh
# If a resource is stuck, remove it from state (doesn't delete from AWS)
pulumi state delete '<resource-urn>'
# Import an existing AWS resource into Pulumi state
pulumi import aws:ec2/securityGroup:SecurityGroup my-sg sg-0123456789