Protected task images¶

A private task's images are as sensitive as its code. Protected task images let a private repository's CI publish them to an ECR repository of its own, <env>-tasks/github.com/<owner>/<repo>, under the repository's code-access rule. A run pulls them with a short-lived login of its own, only if it may use the repository's code.

How it works¶

The lock: each repository's policy denies ecr:* to everyone but Hawk's roles, the break-glass principals, the account root and AWS services. Tags are immutable except buildcache*; untagged images expire after 7 days.
Setup: the broker creates each repository on its first publish, and refuses (409) one that held images before it was pinned, or a pinned one whose policy, lifecycle policy or tag settings differ from Hawk's.
The pin: a hawk-github-repository-id tag holds the GitHub repository's numeric ID, so a repository that later takes the name (a rename or transfer) gets 409.
Creating one first: a principal that can create, tag and push to ECR repositories can create a protected repository before Hawk does, with a matching pin, Hawk's settings and its own images, and publishing then accepts it. Reserve ecr:CreateRepository and ecr:TagResource on <env>-tasks/* for the publishing copy's role (<env>-hawk-task-image-publish-broker), with an SCP, say.
Scanning and replication: keep <env>-tasks/ out of registry scan filters, where the lock refuses Inspector and every scan fails, and out of replication rules, whose copies don't carry the lock.

Opting a repository in¶

Make sure a code-access rule covers github.com/<owner>/<repo> (a * or owner-prefix rule counts), or ask a Hawk admin for one. Publishing is refused unless one does: an unrestricted repository's images belong in the shared one.
Make sure the owner's ID is in hawk:taskImageGithubOwnerIds.
Protect every ref that publishes, as Protecting publish refs describes.
In a workflow that push, workflow_dispatch or schedule starts, replace the AWS role and ECR login with:

on:
  push:
    branches: [main]
  workflow_dispatch: # A branch publishes only if hawk:taskImagePublishRefs allows it.

jobs:
  images:
    runs-on: ubuntu-24.04 # No `environment:`: it changes the token's `sub`, and AWS refuses the role.
    permissions:
      contents: read
      id-token: write # Both OIDC tokens: the one AWS checks and the one the broker checks.
    env: # The stack's outputs (`pulumi stack output`):
      HAWK_REQUESTER_ROLE: arn:aws:iam::123456789012:role/prd-hawk-task-image-requester # task_image_requester_role_arn
      HAWK_TOKEN_BROKER: arn:aws:lambda:us-west-2:123456789012:function:prd-inspect-ai-token_broker-publish:current # task_image_broker_alias_arn
      HAWK_AUDIENCE: https://api.hawk.example.org/images/publish # task_image_github_audience
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false # Keeps the job's token out of .git, so out of the build context.
      - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
        with:
          role-to-assume: ${{ env.HAWK_REQUESTER_ROLE }}
          aws-region: us-west-2 # The stack's region, as in the broker's ARN.
      - name: Log in to the protected task image repository
        shell: bash # With pipefail, so a failed curl stops the step.
        env: # Five publishes run at once: a matrix build's others are throttled, so keep retrying.
          AWS_RETRY_MODE: adaptive
          AWS_MAX_ATTEMPTS: "10"
        run: |
          # Outside the workspace, so never in a build context or artifact, and gone on any exit.
          event="$RUNNER_TEMP/hawk-event.json" response="$RUNNER_TEMP/hawk-response.json"
          trap 'rm -f "$event" "$response"' EXIT
          oidc_token=$(curl -sS --fail-with-body \
            -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
            "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=$HAWK_AUDIENCE" | jq -r .value)
          echo "::add-mask::$oidc_token"
          jq -n --arg authorization "Bearer $oidc_token" \
            '{rawPath: "/images/publish", headers: {authorization: $authorization}}' > "$event"
          aws lambda invoke --function-name "$HAWK_TOKEN_BROKER" \
            --cli-binary-format raw-in-base64-out --payload "file://$event" "$response" > /dev/null
          if [ "$(jq -r .statusCode "$response")" != 200 ]; then cat "$response"; exit 1; fi
          credentials=$(jq -r .body "$response")
          echo "::add-mask::$(jq -r .password <<<"$credentials")"
          # Written straight into a config GitHub empties after the job, so neither a
          # self-hosted runner's default config nor a credential helper keeps it. It has no
          # builders, contexts or other logins: set up buildx, and log in elsewhere, after it.
          export DOCKER_CONFIG="$RUNNER_TEMP/docker"
          echo "DOCKER_CONFIG=$DOCKER_CONFIG" >> "$GITHUB_ENV"
          (umask 077 && mkdir -p "$DOCKER_CONFIG" &&
            jq '{auths: {(.registry): {auth: ("\(.username):\(.password)" | @base64)}}}' \
              <<<"$credentials" > "$DOCKER_CONFIG/config.json")
          echo "DOCKER_IMAGE_REPO=$(jq -r .repository <<<"$credentials")" >> "$GITHUB_ENV"
      - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
      - run: docker buildx bake --push # Tags ${DOCKER_IMAGE_REPO}:<task>-<version>.

Log in just before pushing: the login, for this one repository, lasts three hours.
Triggers: only push, workflow_dispatch and schedule runs publish, since only writers can start them, so move a workflow_run build into such a workflow. Build the commit the run is for (github.sha): checking out a ref from workflow_dispatch inputs publishes code nobody reviewed. Tags are immutable, so where branches may publish, tag their builds <task>-<version>-<branch>, or main can't push a version a branch pushed first.
Other images in the registry: this login replaces any other for the account's registry, so pull FROM or --cache-from <env>/inspect-tasks or tasks-cache with a separate login (its own DOCKER_CONFIG, say), or use public base images.
BuildKit cache: keep it under a mutable buildcache-<task> tag, which no other repository's CI can push to, with --cache-from and --cache-to type=registry,ref=$DOCKER_IMAGE_REPO:buildcache-<task>,mode=max,image-manifest=true,oci-mediatypes=true. Any allowed branch can overwrite it, so give branch builds their own buildcache-<branch> tags and have main import only its own; where main is protected from some writers, restrict hawk:taskImagePublishRefs instead.
Repositories with tasks for the METR task bridge can't opt in yet: they push -info- artifacts with ORAS's ECR backend, which needs AWS credentials, and runs read them with job credentials, which protected repositories deny.
Older images in <env>/inspect-tasks stay readable by every job and the node role until deleted: publish the ones in use again from CI, point tasks at them, and delete them there. Never publish a private image to both.

Protecting publish refs¶

Every push to an allowed ref publishes (main, by default, even where the default branch has another name), and the broker can't see how a ref is protected. So make sure no one writer can get code onto those refs without someone else's review, with an Active GitHub ruleset (new rulesets start Disabled):

Branches: Require a pull request before merging with Required approvals at 2, Dismiss stale pull request approvals when new commits are pushed and Require approval of the most recent reviewable push; plus Restrict creations, Restrict deletions and Block force pushes.
Tags can't require approval: restrict creating, updating and deleting allowed tags to the people who may publish.
Who else can approve or bypass: only people who may publish belong on the bypass list or administer the repository. Keep Allow GitHub Actions to create and approve pull requests off, don't let Copilot approvals count toward merging, install no app that approves pull requests itself (such as renovate-approve), and keep tokens that can approve pull requests or bypass the ruleset out of secrets workflows can read.
Wildcards: a ruleset's * and ? stop at / and IAM's don't, so for refs/heads/release/* target release/**/*, and for refs/tags/v* both v* and v*/**/*.

A repository that can't have an Active ruleset (a private one on GitHub Free) or has no second reviewer can't meet this, so don't opt it in.

Running tasks with protected images¶

Name a protected image by its full repository URL. Runners get the stack's <registry>/<env>-tasks as HAWK_TASK_IMAGES, so one compose file works on every stack (a values file isn't substituted, so it spells the URL out):

services:
  default:
    image: ${HAWK_TASK_IMAGES}/github.com/<owner>/<repo>:<task>-<version>

A run may pull a protected image only if it may fetch the repository's code under the rule Git applies to private repositories: no rule restricts it, or one of the run's tasks, packages or code_sources came from it, as launch checked against the launcher's access. A run can name at most 20 protected repositories, fewer with long names, counting samples a limit leaves out; split an eval set that names more.

A refusal fails the run at startup and says what to change. A broker outage fails it once retries run out: resume the run. A run's login lasts an hour and is renewed every 30 minutes; a restricted run's renewal stops when it completes, so if it's kept alive, its sandboxes can't pull again once the login expires. If renewals keep failing until it expires, new sandboxes can't pull protected images and wait up to Hawk's 24-hour Helm timeout: hawk delete the run. The broker counts logins in TaskImagePullIssued and refusals in TaskImagePullDenied by error_type: invalid, job-type, too-many, identity, unprotected or code; outages aren't counted.

Operator configuration¶

Key	Meaning
`hawk:taskImageGithubOwnerIds`	Owner (organization or user) IDs, as strings, whose repositories may publish; non-empty turns the feature on. `gh api orgs/<org> --jq .id` (or `users/<user>`) finds one.
`hawk:taskImagePublishRefs`	Refs they may publish from, as IAM `StringLike` patterns. Unset: `["refs/heads/main"]`. Every opted-in repository must protect each one: otherwise any writer can publish unreviewed builds, even under a tag `main` uses later.
`hawk:taskImageBreakGlassPrincipals`	IAM role, user or root ARNs that may also use the repositories, matched with `StringNotLike` on `aws:PrincipalArn`. Name a permission set (`.../AWSReservedSSO_BreakGlass_`): `role/` or `role/aws-reserved/sso.amazonaws.com/*` would also let in the node role and read-only SSO roles. Unset: none.

The account needs GitHub's OIDC provider (token.actions.githubusercontent.com, audience sts.amazonaws.com). Every node that runs sandboxes must run Kubernetes 1.35 or later (the deploy checks only hawk:eksK8sVersion), whose kubelet makes a pod without the pull secret re-check with the registry an image the kubelet pulled for another pod, as long as its imagePullCredentialsVerificationPolicy isn't NeverVerify. The default, NeverVerifyPreloadedImages, doesn't re-check images the kubelet didn't pull (preloaded into the node image, or cached before it recorded pulls), and NeverVerifyAllowlistedImages exempts its allowlist, so never preload or allowlist protected images; AlwaysVerify re-checks every image. The feature reserves five Lambda executions for the broker's publishing copy (<env>-inspect-ai-token_broker-publish), so at most five publishes run at once. CI reads the outputs task_image_requester_role_arn, task_image_broker_alias_arn and task_image_github_audience. Dev stacks get dev-<name>-tasks/ repositories, which teardown leaves for you to delete.

Recovery¶

The broker's 409 says what's wrong. Fix it as the root or a break-glass principal the repository's policy names: delete the repository and let CI publish again, or, if only its settings drifted and its policy never admitted anyone beyond the lock's exemptions at the time, restore them in place.

Changing the break-glass list leaves existing policies as they were, still denying added principals and admitting removed ones, and the broker refuses those repositories until they're rewritten. Rewrite them as the root or a principal on both lists (ECR refuses a policy that would shut out its caller):

settings=$(aws lambda get-function-configuration --function-name <env>-inspect-ai-token_broker-publish \
  --query Environment.Variables.TASK_IMAGE_REPOSITORY_SETTINGS --output text)
for repository in $(aws ecr describe-repositories --output text \
  --query "repositories[?starts_with(repositoryName, '<env>-tasks/')].repositoryName"); do
  aws ecr set-repository-policy --repository-name "$repository" --policy-text "$(jq -c .policy <<<"$settings")"
done

Turning the feature off deletes the roles the lock exempts but keeps the repositories, which then admit anyone who creates a role under one of those names, so delete them too.

Revoking pull logins¶

Removing someone from a code-access group doesn't stop their running runs: hawk delete those runs. To end leaked pull logins at once, use IAM's "Revoke active sessions" on the pull role, <env>-hawk-task-image-pull, which within about a minute denies every login issued before then. Runs get fresh logins at their next renewal, within 30 minutes, and new sandboxes may fail to pull until then. A copied job token can still mint logins until it expires, up to an hour after it was issued, so if one may have leaked, revoke again an hour after the run's pods are gone. A node that pulled an image with a revoked login still starts it for any pod given that login, so after the last revocation, replace the nodes the run's sandboxes ran on, or every node, since a run's pods can tolerate any taint, if anyone who can launch runs may have held a leaked login or job token.

When publishing fails¶

The step prints the broker's refusal, which says what to do. Failures it can't explain:

Result	Cause
Assuming the requester role is refused	The owner isn't configured, the ref is outside `hawk:taskImagePublishRefs`, it's a `pull_request` run, the job sets `environment:`, the workflow lacks `id-token: write`, or the organization's GitHub OIDC `sub` template doesn't start with `repo:` and carry `:ref:`.
Invoking is refused	Invoke the alias in `task_image_broker_alias_arn`, in the stack's region. Throttling means five publishes are running: retry.
A push is denied, or the repository doesn't exist	The image names another repository than the login's, or another stack's prefix.

The broker logs each publish and refusal with the repository, workflow, event, ref, commit, actor and run. It counts publishes in TaskImagePublishIssued and refusals in TaskImagePublishDenied by error_type: invalid, owner, ref, event, repository or unrestricted for the run, and tampered, pin-mismatch or unprotected for a repository it won't publish to. Outages aren't refusals; TaskImageRepositoryAccessDenied counts the publishing copy being refused a repository it hasn't set up yet.

Limits¶

Leaked logins: a push login works from anywhere until its deadline. It can pull every image in its repository as well as push new tags and overwrite the mutable buildcache* ones, which ship in the next build. A run's pull login, which its task code can read from the hawk-task-images Secret, also works anywhere until its deadline, and after it still starts the images a node pulled with it, but only for repositories that run may pull anyway.
No alerts: administrators, break-glass principals and the root can widen access by changing the repositories, the registry's settings or the publishing copy, and only CloudTrail records it.
Where runs pull: only images in eval sets' sandbox services (not hawk local) get the login. Others, in sandboxes task code creates, don't, though a pod in the run's sandbox namespace can name its hawk-task-images Secret.
Layer reuse: containerd shares layers across a node's images, so anyone who knows a protected image's layer digests, which CloudTrail's PutImage events record for anyone with cloudtrail:LookupEvents (in AWS's ReadOnlyAccess), can name them in a manifest in any registry and start it where they're cached, without a login. Only separate nodes per code-access group would close this.
Host isolation: Hawk refuses privileged containers and hostPath volumes in sandbox values at every isolation level, but task code can still create such pods in its sandbox namespace with the runner's Kubernetes access. Those pods, or code that escapes its container, can read every image their node cached, other runs' protected ones included, and the pull logins of other runs with pods on that node.