Testing¶

Test Organization¶

Tests are organized by component:

tests/api/ — API server tests
tests/cli/ — CLI command tests
tests/core/ — Core module tests
tests/runner/ — Runner tests
tests/e2e/ — End-to-end tests (requires Minikube)
tests/smoke/ — Smoke tests against live environments

Running Tests¶

# Run all unit tests
pytest

# Run specific package tests (matches CI)
pytest tests/api -n auto -vv
pytest tests/cli -n auto -vv
pytest tests/core -n auto -vv
pytest tests/runner -n auto -vv

# Run E2E tests (requires running Minikube)
pytest --e2e -m e2e -vv

# Run smoke tests
scripts/dev/smoke                          # current stack
scripts/dev/smoke --stack dev-faber        # target a specific stack
scripts/dev/smoke -k test_real_llm         # filter tests by name

Smoke Tests¶

Smoke tests validate a deployed environment by running real evals against real models.

hawk login
scripts/dev/smoke                           # current stack, warehouse included
scripts/dev/smoke --stack staging            # target a specific stack
scripts/dev/smoke --skip-warehouse           # exclude warehouse checks

For validating a dependency bump with them, see Validating a dependency update.

E2E Tests¶

E2E tests require a running Minikube cluster. The happy-path test runs a real eval against OpenAI:

# In your .env:
INSPECT_ACTION_API_RUNNER_SECRET_OPENAI_API_KEY=sk-...
INSPECT_ACTION_API_OPENAI_BASE_URL=https://api.openai.com/v1

Then run:

pytest --e2e -m e2e -vv

Frontend Tests¶

hawk/www has two suites:

pnpm test           # jsdom, fast, the bulk of the coverage
pnpm test:browser   # real Chromium, src/browser only

pnpm test:browser mounts our own components — currently the four ag-grid lists (EvalSetList, ScanList, JobsList, SampleList) — with their data hooks mocked, and asserts the contracts between the grid and our configuration: custom cellRenderers, valueFormatters, getRowId, the object-form rowSelection, defaultColDef.sortable, and the IDatasource that drives SampleList's infinite row model.

A real browser is required because ag-grid only renders rows into a measured viewport; jsdom reports every element as 0×0, so the grid comes out empty and none of the above is exercised.

The goal is drift detection — catching the case where a library changed and our usage did not. That is why the tests render real components rather than synthetic grids: a synthetic grid only re-tests ag-grid's own behaviour, which is upstream's job and tells us nothing about whether EvalSetList still works.

Make new tests earn their place

A test here is only worth its runtime if it fails when our code drifts. Verify a new one by mutating the component it covers — change getRowId, drop a valueFormatter, flip sortable — and confirm it goes red. If it still passes, it is testing the library, not us.

Keep the suite small and the assertions shallow. Notes for adding to it:

Render inside a desktop-sized wrapper (width: 1400). The default test page is ~414px, narrow enough that ag-grid column-virtualises and trailing columns never reach the DOM.
Scope lookups to the grid (.ag-header-cell-text, .ag-row) rather than page-wide getByText, which goes ambiguous against surrounding chrome.

Install the browser once:

pnpm exec playwright install chromium --only-shell

Managed macOS fleets

Playwright's bundled Chromium is ad-hoc signed with no Team ID, and some managed Macs kill it on launch — Playwright then reports Target page, context or browser has been closed. Point it at a locally installed, notarised Chrome instead:

PLAYWRIGHT_CHROMIUM_CHANNEL=chrome pnpm test:browser

Validating a dependency update¶

Which suite actually exercises a bump depends on the ecosystem, so "run the smoke tests" is not always the right answer.

Ecosystem	What validates it
`uv` (Python)	Unit tests, then the smoke suite against a real deployment — this is the most thorough option available
`npm` (`hawk/www`)	`pnpm test`, `pnpm typecheck`, `pnpm build`, and `pnpm test:browser`. Smoke tests never load the browser bundle and add nothing
`docker`	Varies per image — see below. Do not assume E2E or smoke covers it
`github-actions`	Only CI running on the PR itself

Docker bumps are not uniformly covered¶

Dependabot tracks eight Dockerfiles, and CI builds only some of them. Check which image actually changed before assuming a green PR means anything:

Dockerfile	Built in CI by
`.devcontainer/`	`devcontainer.yml`, and E2E runs inside the built image
`hawk/services/modules/<batch>/`, `.../docker_lambda/`	`python-test-batch` / `python-test-lambda`, which build with `target: test` and run the tests inside
`middleman/`, `relay/`	Lint only (`infra-lint.yml`) — not built
`hawk/`, `jumphost/`, `hawk/services/modules/viewer/`	Nothing — first built at deploy time

Smoke tests run whatever images are already deployed, and SMOKE_IMAGE_TAG overrides only the runner. So for the last row, build the image locally and deploy it to a dev stack, or the bump ships unbuilt.

For a Python bump, build a runner image from the branch and point the smoke run at it — no deploy needed:

scripts/dev/build-and-push-runner-image.sh   # prints an image tag
export SMOKE_IMAGE_TAG=<tag>
scripts/dev/smoke --stack dev-<name>          # all tests, no -k filter

Two limits worth knowing:

SMOKE_IMAGE_TAG only replaces the runner image. A bump to an api-group dependency is not exercised until the branch is deployed to a dev stack with pulumi up.
The stg-smoke CI environment only permits main, so smoke.yml cannot be dispatched against a PR branch. Run locally against a dev stack instead.

For full smoke coverage, install skopeo (otherwise the requires_task_images tests skip silently) and leave the warehouse enabled (for requires_warehouse).

Also check for over-broad mocks after any upgrade. Patching a library constructor (aioboto3.Session.client) rather than our own factory (my_module.aws_clients.get_s3_client) breaks whenever a dependency changes how it reaches external services — a previously-passing test failing with MagicMock can't be used in 'await' expression is the signature. See the mock scoping guidance in hawk/tests/CLAUDE.md.

Testing Tools¶

Tool	Purpose
`pytest-xdist`	Parallel test execution (`-n auto`)
`pytest-asyncio`	Async test support (auto mode)
`pytest-mock`	General mocking
`pyfakefs`	Filesystem mocking
`moto`, `pytest-aioboto3`	AWS mocking
`testcontainers[postgres]`	PostgreSQL containers
`time-machine`	Time mocking

Code Quality Checks¶

Must pass before completion:

ruff check .                    # linting
ruff format . --check           # format check
basedpyright .                  # type checking