Testing¶

Test Organization¶

Tests are organized by component:

tests/api/ — API server tests
tests/cli/ — CLI command tests
tests/core/ — Core module tests
tests/janitor/ — Kubernetes janitor tests
tests/runner/ — Runner tests
tests/test_e2e.py — End-to-end tests (requires Minikube)
tests/test_smoke_diagnostics.py — Smoke-diagnostic unit tests
tests/smoke/ — Smoke tests against live environments
services/modules/*/tests/ — Lambda and Batch service tests

Running Tests¶

Run the pytest commands from hawk/. Run scripts/dev/smoke from the repository root.

# Run all unit tests
pytest

# Run specific package tests (matches CI)
pytest tests/api -n auto -vv
pytest tests/cli -n auto -vv
pytest tests/core -n auto -vv
pytest tests/janitor -n auto -vv
pytest tests/runner -n auto -vv

# Run the smoke-diagnostic unit tests (the runner CI leg also runs these)
pytest tests/test_smoke_diagnostics.py -n auto -vv

# Run E2E tests (requires running Minikube)
pytest --e2e -m e2e tests/test_e2e.py -vv --timeout=600

Smoke Tests¶

Smoke tests validate a deployed environment by running real evals against real models.

Run them from the repository root after logging Pulumi into the deployment's S3 backend. The smoke wrapper resolves the selected stack's API URL, but login is interactive and must target that same API first:

export AWS_PROFILE="<profile-for-the-target-account>"
export PULUMI_FALLBACK_TO_STATE_SECRETS_MANAGER=true
pulumi login "s3://<state-bucket>?region=<region>&awssdk=v2"
STACK="<stack>"
API_URL="$(pulumi stack output api_url -s "$STACK")"
HAWK_API_URL="$API_URL" hawk login
scripts/dev/smoke --stack "$STACK"                   # warehouse included
scripts/dev/smoke --stack "$STACK" --skip-warehouse  # exclude warehouse checks
scripts/dev/smoke --stack "$STACK" -k test_real_llm  # filter tests by name

The selected AWS credentials must reach the stack's account. Use hawk login --no-browser in a devcontainer, SSH session, or other headless environment.

For validating a dependency bump with them, see Validating a dependency update.

E2E Tests¶

By default, E2E tests run entirely locally with Minikube and require no METR account, AWS credentials, registry login, or LLM API key. Run these commands inside the devcontainer, starting from the repository root:

cd hawk
uv sync --all-extras --all-groups --locked
source .venv/bin/activate
../scripts/dev/start-minikube.sh
set -a
source .env
set +a
pytest --e2e -m e2e tests/test_e2e.py -vv --timeout=600

The startup script creates hawk/.env from .env.example if needed, builds the API and runner with the public python:3.13-slim-trixie base image, and logs in through the bundled local Dex provider. Local Compose and Minikube builds use this public base by default; production builds retain the Docker Hardened Images base, whose registry requires a Docker login. Internet access is still needed to download public images and Python packages.

The E2E evaluations explicitly request 2Gi of runner memory, so they fit a small local cluster even when hawk/.env retains the general 16Gi default. CI also uses 2Gi for its test environment.

CI uses the same trust decision for Batch/E2E images and the E2E model:

Run	Python base	E2E model
`main` pushes and PR branches in `METR/hawk`, except Dependabot	`dhi.io/python:3.13-dev`	`openai/gpt-4o-mini`
Fork and Dependabot PRs	`python:3.13-slim-trixie`	`mockllm/model`
Local default	`python:3.13-slim-trixie`	`mockllm/model`

The Python image defaults are pinned to multi-architecture digests in CI, Compose, and the Minikube setup script. Update the matching pins together when upgrading the images; the tags above are shown without digests for readability.

Trusted CI keeps Docker Hub and DHI authentication, the private GHCR devcontainer cache, and the existing OpenAI API key. Fork and Dependabot runs need none of those secrets and build the devcontainer from public sources. Forks run the same tests after any required GitHub workflow approval.

To test DHI locally, run this from hawk/ with the virtual environment active:

docker login dhi.io
export PYTHON_BASE_IMAGE=dhi.io/python:3.13-dev@sha256:2e34fb87972c33e06222decf60d07646e6b874faaf6b1820f198cca70a5dc7b1
../scripts/dev/start-minikube.sh

The image override is independent of HAWK_E2E_MODEL; local tests still use the mock unless you select the live model below.

The tests submit evals through the real CLI and API, run Kubernetes jobs, and verify the evaluation logs in MinIO. They install the pinned public inspect_evals task and, by default, use Inspect's deterministic mockllm/model provider. Both model modes execute the task's solver and scorer. The secret test passes dummy values and verifies they reach the job's Kubernetes Secret; this test always uses mockllm/model. The eval task does not require a separate task sandbox. Use the smoke suite for broader coverage of real model providers against a deployed environment.

To run the real OpenAI eval locally, add your own key and the direct OpenAI endpoint to hawk/.env:

HAWK_API_RUNNER_SECRET_OPENAI_API_KEY=sk-...
HAWK_API_OPENAI_BASE_URL=https://api.openai.com/v1

Then, from hawk/ with the virtual environment active, recreate the API so the runner receives the key and select the live model explicitly:

docker compose up -d --force-recreate api
set -a
source .env
set +a
HAWK_E2E_MODEL=openai/gpt-4o-mini pytest --e2e -m e2e tests/test_e2e.py -vv --timeout=600

HAWK_E2E_MODEL supports mockllm/model (the default) and openai/gpt-4o-mini. The latter makes real, billable model requests.

The scan E2E test remains skipped because it hangs in CI; it is not part of the credential-free eval coverage.

Frontend Tests¶

hawk/www has two suites:

# From hawk/www:
pnpm test           # jsdom, fast, the bulk of the coverage
pnpm test:browser   # real Chromium, src/browser only

pnpm test:browser mounts our own components — currently the four ag-grid lists (EvalSetList, ScanList, JobsList, SampleList) — with their data hooks mocked, and asserts the contracts between the grid and our configuration: custom cellRenderers, valueFormatters, getRowId, the object-form rowSelection, defaultColDef.sortable, and the IDatasource that drives SampleList's infinite row model.

A real browser is required because ag-grid only renders rows into a measured viewport; jsdom reports every element as 0×0, so the grid comes out empty and none of the above is exercised.

The goal is drift detection — catching the case where a library changed and our usage did not. That is why the tests render real components rather than synthetic grids: a synthetic grid only re-tests ag-grid's own behaviour, which is upstream's job and tells us nothing about whether EvalSetList still works.

Make new tests earn their place

A test here is only worth its runtime if it fails when our code drifts. Verify a new one by mutating the component it covers — change getRowId, drop a valueFormatter, flip sortable — and confirm it goes red. If it still passes, it is testing the library, not us.

Keep the suite small and the assertions shallow. Notes for adding to it:

Render inside a desktop-sized wrapper (width: 1400). The default test page is ~414px, narrow enough that ag-grid column-virtualises and trailing columns never reach the DOM.
Scope lookups to the grid (.ag-header-cell-text, .ag-row) rather than page-wide getByText, which goes ambiguous against surrounding chrome.

Install the browser once:

pnpm exec playwright install chromium --only-shell

Managed macOS fleets

Playwright's bundled Chromium is ad-hoc signed with no Team ID, and some managed Macs kill it on launch — Playwright then reports Target page, context or browser has been closed. Point it at a locally installed, notarised Chrome instead:

PLAYWRIGHT_CHROMIUM_CHANNEL=chrome pnpm test:browser

Validating a dependency update¶

Which suite actually exercises a bump depends on the ecosystem, so "run the smoke tests" is not always the right answer.

Ecosystem	What validates it
`uv` (Python)	Unit tests, then the smoke suite against a real deployment — this is the most thorough option available
`npm` (`hawk/www`)	`pnpm test`, `pnpm typecheck`, `pnpm build`, and `pnpm test:browser`. Smoke tests never load the browser bundle and add nothing
`docker`	Varies per image — see below. Do not assume E2E or smoke covers it
`github-actions`	Only CI running on the PR itself

Docker bumps are not uniformly covered¶

Dependabot tracks eight Dockerfiles, and CI builds only some of them. Check which image actually changed before assuming a green PR means anything:

Dockerfile	Built in CI by
`.devcontainer/`	`devcontainer.yml`, and E2E runs inside the built image
`hawk/services/modules/<batch>/`, `.../docker_lambda/`	`python-test-batch` / `python-test-lambda`, which build with `target: test` and run the tests inside
`middleman/`, `relay/`	Lint only (`infra-lint.yml`) — not built
`hawk/`, `jumphost/`, `hawk/services/modules/viewer/`	Nothing — first built at deploy time

Smoke tests run whatever images are already deployed, and SMOKE_IMAGE_TAG overrides only the runner. So for the last row, build the image locally and deploy it to a dev stack, or the bump ships unbuilt.

For runner-only iteration without AWS, use the local Minikube workflow. For the more thorough validation of a Python bump against deployed services, build a runner image from the branch and point a dev stack's smoke run at it. This does not require deploying the whole branch. Complete the Smoke Tests setup above in the same shell, then authenticate Docker to the stack's ECR and run:

IMAGE_TAG=my-tag
ENVIRONMENT="$(pulumi stack output env -s "$STACK")"
AWS_REGION="$(pulumi stack output region -s "$STACK")"
(
  cd hawk
  PULUMI_STACK="$STACK" \
    ENVIRONMENT="$ENVIRONMENT" AWS_REGION="$AWS_REGION" \
    ../scripts/dev/build-and-push-runner-image.sh "$IMAGE_TAG"
)
SMOKE_IMAGE_TAG="$IMAGE_TAG" scripts/dev/smoke --stack "$STACK"  # all tests, no -k filter

Two limits worth knowing:

SMOKE_IMAGE_TAG only replaces the runner image. A bump to an api-group dependency is not exercised until the branch is deployed to a dev stack with pulumi up.
The stg-smoke CI environment only permits main, so smoke.yml cannot be dispatched against a PR branch. Run locally against a dev stack instead.

For full smoke coverage, install skopeo (otherwise the requires_task_images tests skip silently) and leave the warehouse enabled (for requires_warehouse).

Also check for over-broad mocks after any upgrade. Patching a library constructor (aioboto3.Session.client) rather than our own factory (my_module.aws_clients.get_s3_client) breaks whenever a dependency changes how it reaches external services — a previously-passing test failing with MagicMock can't be used in 'await' expression is the signature. See the mock scoping guidance in hawk/tests/CLAUDE.md.

Testing Tools¶

Tool	Purpose
`pytest-xdist`	Parallel test execution (`-n auto`)
`pytest-asyncio`	Async test support (auto mode)
`pytest-mock`	General mocking
`pyfakefs`	Filesystem mocking
`moto`, `pytest-aioboto3`	AWS mocking
`testcontainers[postgres]`	PostgreSQL containers
`time-machine`	Time mocking

Code Quality Checks¶

Must pass before completion. Run the repo's pre-commit hooks — the same checks CI's Lint job runs (see Code Quality for setup):

pre-commit run --all-files