Managing Your Deployment¶
Day-to-day Commands¶
pulumi up # deploy changes
pulumi preview # preview without deploying
pulumi stack output --json # view outputs (API URL, DB endpoint, etc.)
pulumi refresh # sync Pulumi state with actual AWS resources
Updating Hawk¶
Pull the latest code and redeploy:
Database migrations run automatically during deployment.
Updating Inspect AI or Scout¶
Use the release preparation script:
scripts/ops/prepare-release.py --inspect-ai 0.3.50 # specific PyPI version
scripts/ops/prepare-release.py --inspect-ai abc123 # specific git commit
scripts/ops/prepare-release.py --inspect-scout 0.2.10 # update Scout
Then deploy and run smoke tests to validate.
Runner Default Environment Variables¶
Pulumi creates <env>/hawk/runner-default-env containing {}. Write key/value pairs into it to inject env vars into every runner job (e.g. shared tokens not routed through Middleman):
aws secretsmanager put-secret-value \
--secret-id <env>/hawk/runner-default-env \
--secret-string '{"WANDB_API_KEY": "..."}'
Runtime values (auth tokens, Sentry, provider secrets) and user-supplied --secret overrides take precedence. Cache TTL: ~5 min.
Smoke Tests¶
Validate that a deployed environment is working end-to-end:
hawk login
scripts/dev/smoke # test current stack
scripts/dev/smoke --stack my-org # test a specific stack
scripts/dev/smoke --warehouse # include database checks
scripts/dev/smoke -k test_real_llm # run a specific test
Smoke tests submit real evals against real models and verify results end up in the warehouse and viewer.
After updating Inspect AI or Scout dependencies:
Tearing Down¶
Warning
Always wait for pulumi destroy to finish before running stack rm. Running stack rm first will orphan AWS resources in your account.
Teardown caveats
- Protected resources: Some resources (Secrets Manager secrets, S3 buckets) are protected in Pulumi state to prevent accidental deletion. If
pulumi destroyskips resources, unprotect them first:pulumi state unprotect '<urn>' - Non-empty S3 buckets: S3 buckets must be empty before deletion. Empty them with
aws s3 rm s3://<bucket> --recursiveif destroy fails.
Cleaning up bootstrap resources¶
pulumi destroy does not remove the S3 state bucket or KMS key you created during Quick Start step 3 — those live outside any stack. The KMS key costs $1.00/month (prorated hourly) until you schedule it for deletion; billing stops as soon as it's scheduled. Run these after pulumi stack rm completes, using the same region as bootstrap:
# Delete the state bucket
aws s3 rb s3://<state-bucket-name> --force
# Schedule the KMS key for deletion (minimum 7 days, default 30)
KEY_ID=$(aws kms describe-key --key-id alias/pulumi-secrets --region <region> \
--query 'KeyMetadata.KeyId' --output text)
aws kms delete-alias --alias-name alias/pulumi-secrets --region <region>
aws kms schedule-key-deletion --key-id "$KEY_ID" --pending-window-in-days 7 --region <region>
Versioned state bucket
If you enabled versioning on the state bucket (Pulumi recommends it for state recovery), aws s3 rb --force will not remove object versions or delete markers and fails with BucketNotEmpty. Empty the bucket first using the S3 console's Empty action or aws s3api delete-objects + aws s3api list-object-versions, then re-run rb.
Troubleshooting¶
Pulumi State Issues¶
# See what Pulumi thinks exists vs what's actually in AWS
pulumi refresh
# If a resource is stuck, remove it from state (doesn't delete from AWS)
pulumi state delete '<resource-urn>'
# Import an existing AWS resource into Pulumi state
pulumi import aws:ec2/securityGroup:SecurityGroup my-sg sg-0123456789