Skip to content

Container Images (agent reference)

Reference for AI agents and contributors changing how Hawk's images are built. The operator-facing summary is Container Images; keep both pages in sync when you change how images are built, tagged, cached or pinned.

Every Hawk-owned image is a pulumi_docker_build.Image built from the checkout on every pulumi up. There is no prebuilt-image or image-URI override.

sequenceDiagram
    autonumber
    participant P as Pulumi
    participant D as Docker builder
    participant I as Internet
    participant E as ECR
    participant S as Services
    P->>E: Create repositories
    P->>P: Check each image for changes
    P->>D: Build changed images
    D->>I: Pull base images, clone inspect_ai
    D->>E: Push image and build cache
    P->>S: Point services at the new digest

Where things live

  • infra/lib/docker_build.py
  • make_ecr_repo(): the aws.ecr.Repository, plus an aws.ecr.LifecyclePolicy when lifecycle_rules is passed. Defaults: force_delete=True, MUTABLE tags, no scanning.
  • ECRImageArgs: wraps docker_build.ImageArgs. Sets push=True, one tag (<repo>:latest), ECR registry auth from one shared get_authorization_token_output(), platforms=[architecture.docker_platform], and the build cache (below).
  • Lifecycle rule helpers: keep_last_n_any_rule, expire_any_after_days_rule, expire_untagged_after_days_rule.
  • infra/lib/architecture.py: CpuArchitecture and DEFAULT_CPU_ARCHITECTURE (arm64), read from hawk:cpuArchitecture. One platform per stack; no multi-arch manifests.
  • Per-image definitions (Dockerfile → target):
Image Defined in Dockerfile → target
api infra/hawk/image.py (HawkImage) hawk/Dockerfile → api
runner infra/hawk/ecr.py (HawkEcr) hawk/Dockerfile → runner
janitor infra/k8s/janitor.py hawk/Dockerfile → janitor
viewer infra/hawk/viewer/image.py (ViewerImage) hawk/services/modules/viewer/Dockerfile
Lambdas infra/hawk/lambdas.py (DockerLambda) hawk/services/modules/docker_lambda/Dockerfile → prod, SERVICE_NAME build arg
eval-log-importer infra/hawk/eval_log_importer.py hawk/services/modules/eval_log_importer/Dockerfile → prod
sample-editor infra/hawk/sample_editor.py hawk/services/modules/sample_editor/Dockerfile → prod
middleman infra/core/middleman.py middleman/Dockerfile → prod, named context hawk
relay infra/hawk/relay.py relay/Dockerfile → prod, named context hawk
jumphost infra/core/jumphost.py jumphost/Dockerfile

DockerLambda users: token_broker.py, job_status_updated.py, scan_importer.py, and eval_log_reader.py (only with hawk:enableS3ObjectLambda). Only runner and janitor pass hawk:basePythonImage as the PYTHON_BASE_IMAGE build arg; the other Dockerfiles use their own pinned default. The viewer's VITE_* build args make its image stack-specific.

How rebuilds, caching and pinning work

  • Rebuild decision: the provider's Diff compares contextHash (a hash of the context, Dockerfile and named contexts after .dockerignore) and the other inputs. Registry password changes are ignored. A changed hash is an update, not a replace.
  • Named contexts: the provider hashes a named context with the primary context's .dockerignore. So middleman/.dockerignore and relay/.dockerignore must not exclude the Hawk files those Dockerfiles copy. infra/tests/test_docker_build_contexts.py pins this.
  • Cache: outside GitHub Actions, cache_from/cache_to use <repo>:buildcache in the same ECR repo (mode=max, image_manifest=True, oci_media_types=True). With GITHUB_ACTIONS=true they use the GHA cache, scoped by repo name. A new stack starts with an empty cache.
  • Pinning: consumers use f"{repo_url}@{image.digest}" (ECS task definitions, Lambda image_uri, Batch, janitor, the API's default runner image). Never point a consumer at :latest.
  • buildOnPreview: not set, so the provider default (true) applies and pulumi preview builds images. CI's scripts/ci/filter_pulumi_preview.py strips the image churn this causes from preview comments.

ECR lifecycle rules

  • api, viewer, Lambdas, eval-log-importer, sample-editor, middleman, relay: keep_last_n_any_rule(10).
  • runner: expire_any_after_days_rule(StackConfig.runner_image_retention_days) (hawk:expectedLongestRunDays, default 30, plus 7). It is age-based on purpose; see the comment at the top of infra/hawk/ecr.py.
  • janitor and jumphost: inline policies in their own files, not the helpers.
  • inspect-tasks, tasks-cache, custom-runners and viewer plug-in repos are created here but nothing is built into them at deploy time.

Pitfalls

  • Shared uv cache mount: the Python Dockerfiles use --mount=type=cache,target=/root/.cache/uv with no id or sharing, so parallel builds on one builder share it and can wait on uv's cache lock. The apt and pnpm mounts use sharing=locked.
  • Changing the Docker host replaces images: host (default DOCKER_HOST) is config on the default docker-build provider, not an Image input. pulumi-go-provider replaces an infer provider on any config change except version, and the engine then replaces every Image that provider owns. Switching the default buildx builder changes neither.
  • A new checkout path rebuilds every image: context.location and dockerfile.location are absolute (infra/lib/repo_paths.py resolves the checkout). Moving the repo or deploying from another machine shows them as diffs. They are an update (rebuild), not a replace.
  • Registry logins come from where Pulumi runs: the provider reads Docker Hub and dhi.io logins from the Docker config on the machine running Pulumi and passes them to the builder, even a remote one. Only the ECR login is passed in registries.
  • Replace with an unchanged digest deletes the live image: replace is create-before-delete, and the provider's Delete removes <repo>@<old digest> from the registry. If the rebuild produced the same digest, that deletes the image the new resource and its consumers point at, and pulls fail. No image sets retain_on_delete or delete_before_replace today.
  • Parallelism: images depend only on their repo,1 so they all build at once, bounded only by pulumi up --parallel. Builds have no per-image timeout.

How to check a change

  • pulumi preview --diff -s <stack>: an image update is a rebuild; a replace on an Image resource needs a reason before you apply it.
  • uv run pytest infra/tests/test_docker_build_contexts.py after touching a Dockerfile's COPY --from=hawk lines or a .dockerignore.
  • Per-image build-and-push time is in the pulumi up output, on lines like docker-build:index:Image <name> updated (95s). The production workflow's logs (.github/workflows/pulumi-production.yml) carry the same lines for METR's deploys.

  1. Except the viewer, whose VITE_OIDC_* build args (infra/hawk/viewer/image.py) wait for Cognito when the stack creates it. ↩