Skip to content

Managing Your Deployment

Day-to-day Commands

Run these commands from the repository root. Log in to the deployment's S3 backend and choose the target stack once per shell session:

export AWS_PROFILE="<profile-for-the-target-account>"
export PULUMI_FALLBACK_TO_STATE_SECRETS_MANAGER=true
pulumi login "s3://<state-bucket>?region=<region>&awssdk=v2"
STACK="<stack>"
ENVIRONMENT="$(pulumi stack output env -s "$STACK")"
AWS_REGION="$(pulumi stack output region -s "$STACK")"

pulumi up -s "$STACK"                     # deploy changes
pulumi preview -s "$STACK"                # preview without deploying
pulumi stack output --json -s "$STACK"    # view outputs (API URL, DB endpoint, etc.)
pulumi refresh -s "$STACK"                # sync Pulumi state with actual AWS resources

The selected AWS credentials must have access to the target stack's account; pulumi login selects the state backend, not the AWS identity used for changes.

Updating Hawk

Pull the latest code and redeploy:

git pull
pulumi up -s "$STACK"

Database migrations run automatically during deployment.

Updating Inspect AI or Scout

Use the release preparation script:

scripts/ops/prepare-release.py --inspect-ai 0.3.50    # specific PyPI version
scripts/ops/prepare-release.py --inspect-ai abc123     # specific git commit
scripts/ops/prepare-release.py --inspect-scout 0.2.10  # update Scout

Then deploy and run smoke tests to validate.

Runner Default Environment Variables

Pulumi creates <env>/hawk/runner-default-env containing {}. Write key/value pairs into it to inject env vars into every runner job (e.g. shared tokens not routed through Middleman):

aws secretsmanager put-secret-value \
  --secret-id "${ENVIRONMENT}/hawk/runner-default-env" \
  --secret-string '{"WANDB_API_KEY": "..."}' \
  --region "$AWS_REGION"

Runtime values (auth tokens, Sentry, provider secrets) and user-supplied --secret overrides take precedence. Cache TTL: ~5 min.

Smoke Tests

Validate that a deployed environment is working end-to-end:

API_URL="$(pulumi stack output api_url -s "$STACK")"
HAWK_API_URL="$API_URL" hawk login
scripts/dev/smoke --stack "$STACK"                   # warehouse included
scripts/dev/smoke --stack "$STACK" --skip-warehouse  # exclude warehouse checks
scripts/dev/smoke --stack "$STACK" -k test_real_llm  # filter tests by name
scripts/dev/smoke --stack "$STACK" --refresh-stack   # refresh cached stack outputs

Smoke tests submit real evals against real models and verify results end up in the warehouse and viewer. Add --no-browser to the stack-targeted login command in a headless environment.

Tearing Down

The easiest path is the teardown script, which automates the whole sequence below (confirmation prompt, deletion-guard removal, bounded Karpenter drain, destroy, stack removal):

scripts/dev/teardown.sh "$STACK"

Expect a full teardown to take well over an hour — EKS, RDS, NAT, and VPC deletion alone commonly run ~1h15m; that's AWS-side deletion time, not something Hawk can speed up.

To tear down manually, run these two phases in order:

# Phase 1: remove the deletion guards declaratively.
# protectResources=false flips — in one `pulumi up` — the pulumi protect flags,
# the ALB's deletion protection, force_destroy on S3 buckets, and force_delete
# on ECR repos. (`pulumi state unprotect` alone is NOT enough: it clears the
# state flags but leaves those AWS-side guards baked into the resources.)
pulumi config set hawk:protectResources false -s "$STACK"
pulumi up --yes -s "$STACK"

# Phase 2: destroy and remove the stack.
PULUMI_K8S_DELETE_UNREACHABLE=true pulumi destroy --yes -s "$STACK" &&
  pulumi stack rm -s "$STACK"     # only after destroy succeeds

Phase 1 is a regular pulumi up, so it needs the same prerequisites as a deploy (Docker running and logged in for the image builds). If the deployment is too broken for pulumi up to succeed, fall back to pulumi state unprotect --all --yes -s "$STACK" and expect to handle the ALB/S3/ECR guards manually (see Troubleshooting teardown).

Warning

Always wait for pulumi destroy to finish before running stack rm. Running stack rm first will orphan AWS resources in your account. Don't pipe long-running destroys through tee — it masks Pulumi's non-zero exit code as success.

Troubleshooting teardown

Each of these blocks or hangs a pulumi destroy; phase 1 prevents most of them, and the rest have a manual fix.

Unreachable EKS provider

If the cluster is gone or the provider's auth in state has gone stale, pulumi destroy fails early with configured Kubernetes cluster is unreachable. PULUMI_K8S_DELETE_UNREACHABLE=true lets Pulumi drop those Kubernetes resources from state and continue.

Karpenter NodeClaims hang the destroy silently

Destroy deletes the NodePools, but a NodeClaim's finalizer waits for its node to drain — and a node holding pods that refuse eviction (e.g. runner/sandbox pods carry karpenter.sh/do-not-disrupt) never finishes. The destroy then sits with no error, potentially for 40+ minutes. teardown.sh avoids this by draining NodeClaims with a bounded wait before the destroy; if you hit it manually, force-finalize the stuck NodeClaim:

# find the stuck claim + its EC2 instance
EKS_CLUSTER_NAME="$(pulumi stack output eks_cluster_name -s "$STACK")"
aws eks update-kubeconfig --name "$EKS_CLUSTER_NAME" --region "$AWS_REGION"
kubectl get nodeclaims
aws ec2 terminate-instances --instance-ids <instance-id> --region "$AWS_REGION"
kubectl patch nodeclaim <name> -p '{"metadata":{"finalizers":null}}' --type=merge

ALB refuses deletion (OperationNotPermitted)

The ALB ships with deletion protection while protectResources is on, and it blocks the VPC teardown behind it. Phase 1 turns it off; if you skipped phase 1, disable it out-of-band:

ALB_ARN="$(pulumi stack output alb_arn -s "$STACK")"
aws elbv2 modify-load-balancer-attributes --load-balancer-arn "$ALB_ARN" \
  --attributes Key=deletion_protection.enabled,Value=false \
  --region "$AWS_REGION"

Non-empty S3 buckets and ECR repos

BucketNotEmpty: phase 1 sets force_destroy so this shouldn't happen; if you skipped it, empty the bucket first — versioned buckets need all object versions and delete markers removed, not just aws s3 rm --recursive. RepositoryNotEmptyException: likewise covered by phase 1 (force_delete); otherwise delete the images first with aws ecr batch-delete-image.

Redeploying to the same account within 30 days

If the stack was destroyed while protectResources was still true, its Secrets Manager secrets were deleted with a 30-day recovery window, and the next pulumi up fails with "secret is already scheduled for deletion" on every <env>/… name. Phase 1 avoids this (with protectResources: false secrets delete immediately); to recover after the fact, force-purge the colliding secrets and re-run the deploy:

aws secretsmanager list-secrets \
  --query "SecretList[?starts_with(Name, '${ENVIRONMENT}/')].Name" --output text \
  --region "$AWS_REGION" | tr '\t' '\n' |
  xargs -I{} aws secretsmanager delete-secret --secret-id {} \
    --force-delete-without-recovery --region "$AWS_REGION"

Recovering from an interrupted destroy

An interrupted pulumi destroy (Ctrl-C, crash, dropped connection) can leave a stale lock and pending operations, so pulumi stack rm fails with a lock error or "still has resources". Don't use pulumi stack rm --force to get past it — that removes the stack from state without deleting the resources (the same orphaning the warning above describes).

# Release the stale lock. The lock error names the host + pid holding it;
# confirm that process is gone before cancelling (if the host isn't yours,
# ask its owner) — cancelling a live operation can corrupt state.
# https://www.pulumi.com/docs/iac/cli/commands/pulumi_cancel/
# "already completed" means there was no lock — ignore it.
pulumi cancel -s "$STACK"

# Pending deletes resolve automatically; pending creates prompt for a choice.
pulumi refresh -s "$STACK"

pulumi destroy -s "$STACK"

See Recovering from Interrupted Updates.

Cleaning up bootstrap resources

pulumi destroy does not remove the resources you created manually before the first deploy — those live outside any stack: the S3 state bucket and KMS key from Quick Start step 3, and, if you pre-created/delegated one, the Route 53 public hosted zone for hawk:publicDomain. The KMS key costs $1.00/month (prorated hourly) until you schedule it for deletion; billing stops as soon as it's scheduled. These resources may be shared by multiple stacks: remove them only after the final dependent stack is gone, never as routine dev-stack cleanup. Use the same region as bootstrap:

# Delete the state bucket
aws s3 rb s3://<state-bucket-name> --force

# Schedule the KMS key for deletion (minimum 7 days, default 30)
KEY_ID=$(aws kms describe-key --key-id alias/pulumi-secrets --region <region> \
  --query 'KeyMetadata.KeyId' --output text)
aws kms delete-alias --alias-name alias/pulumi-secrets --region <region>
aws kms schedule-key-deletion --key-id "$KEY_ID" --pending-window-in-days 7 --region <region>

# Delete the public hosted zone (must contain only its NS + SOA records;
# delete any other records first)
aws route53 delete-hosted-zone --id <zone-id>

Finally, remove the NS delegation for the zone at your registrar or parent DNS provider (e.g. the four NS records you added in Cloudflare) — otherwise it dangles, pointing at a zone that no longer exists.

Versioned state bucket

If you enabled versioning on the state bucket (Pulumi recommends it for state recovery), aws s3 rb --force will not remove object versions or delete markers and fails with BucketNotEmpty. Empty the bucket first using the S3 console's Empty action or aws s3api delete-objects + aws s3api list-object-versions, then re-run rb.

Troubleshooting

Pulumi State Issues

# See what Pulumi thinks exists vs what's actually in AWS
pulumi refresh -s "$STACK"

# If a resource is stuck, remove it from state (doesn't delete from AWS)
pulumi state delete '<resource-urn>' -s "$STACK"

# Import an existing AWS resource into Pulumi state
pulumi import aws:ec2/securityGroup:SecurityGroup my-sg sg-0123456789 -s "$STACK"

Redeploying a Single Resource

pulumi up -s "$STACK" --target "urn:pulumi:${STACK}::hawk::..."
# Tip: run `pulumi stack export -s "$STACK"` to find resource URNs

GPU Operator Deploy Failures (NGC Egress / Pending Operation Jam)

The GPU operator is enabled by default (hawk:enableGpuOperator: "true"). CPU-only stacks can opt out:

pulumi config set hawk:enableGpuOperator false -s "$STACK"

Common failure modes:

  • 403 Forbidden from helm.ngc.nvidia.com — NGC's WAF is blocking the Helm pull. Set hawk:enableGpuOperator: "false" until egress to NGC is fixed.
  • context deadline exceeded on gpu-operator-release — NGC is responding too slowly and the Helm release times out. Same workaround: disable the operator until NGC recovers.
  • Pending-operation jam — a previous deploy failed mid-way through the GPU operator Helm release; see the recovery path below.

Recovery path for stacks already stuck in a pending-operation jam:

If a previous deploy failed mid-way through the GPU operator Helm release, Pulumi may show a warning about pending operations on the next pulumi up. Fix with:

# Reconcile Pulumi state with what actually exists in AWS
pulumi refresh -s "$STACK"

# Then redeploy — GPU resources will be skipped if enableGpuOperator is false
pulumi up -s "$STACK"

Drain GPU nodes before disabling on a live stack

Disabling the operator on a stack currently running GPU pods requires draining those nodes first. The operator's depends_on chain makes Pulumi destroy the operator (DaemonSets, device plugin, DCGM) before the NodePools, so any scheduled GPU pods lose their drivers mid-eviction.