Skip to content

Contributing

Developer Setup

There are two ways to run Hawk locally. Choose the workflow that matches the change you're making:

  • Full Dev Stack for API and viewer development with live reload. Use this for most application and frontend work.
  • Local Minikube Setup for the complete local platform, including MinIO, PostgreSQL, the API, and eval runners on Minikube. Use this when the change affects how evals actually execute.

Commit signing

All commits to this repo must be signed. Set up SSH commit signing once (global config applies to every clone and worktree):

git config --global commit.gpgsign true
git config --global gpg.format ssh
# point at your SSH public key (this is the common default path)
git config --global user.signingkey ~/.ssh/id_ed25519.pub

Then add that same public key to GitHub as a Signing Key (Settings → SSH and GPG keys → New SSH key → key type "Signing Key"). A key added only as an Authentication key still signs locally but won't show as Verified on GitHub.

Confirm a commit is signed with git cat-file -p <sha> (look for a gpgsig header) or git log --show-signature; on GitHub it shows a "Verified" badge.

Full Dev Stack: API, Viewer, Live Reload

For developing with hot reload across the full stack:

Terminal 1: Viewer Dev Server

cd hawk/www
pnpm install
VITE_API_BASE_URL=http://localhost:8080 pnpm dev

Terminal 2: API Server

Run from the repo root:

# Point an env file at a deployed stack (DB, S3, OIDC, etc.):
uv run python scripts/dev/generate-env.py <stack> --api > hawk/.env
# ...or, for fully local development: cp hawk/.env.example hawk/.env

scripts/dev/api   # serves http://localhost:8080 with live reload

scripts/dev/api loads hawk/.env (override with HAWK_ENV_FILE) and runs the server in the app's project env. Extra args pass through to fastapi dev, e.g. scripts/dev/api --port 9000.

Optional: Embedded Viewer Library Watch Mode

Hawk consumes packaged Inspect AI and Inspect Scout viewers, so ordinary Hawk frontend development does not require either repository. If a change is inside one of those embedded viewer libraries, follow the cross-repository setup. It covers cloning the relevant repository at the revision pinned by Hawk, initializing its ts-mono submodule, and temporarily linking the local viewer package into hawk/www.

Code Quality

Linting and type checking are defined by the repo's pre-commit hooks (.pre-commit-config.yaml). CI's Lint job runs the same hooks, so run them locally rather than invoking the tools by hand.

Install pre-commit:

uv tool install pre-commit --with pre-commit-uv
pnpm --dir hawk/www install   # once, so the eslint/prettier/tsc hooks can run

Run linting + type checks (requires all python and js toolchains to be installed locally):

pre-commit run --all-files

Unit tests are not part of the hooks — run pytest in the package you're changing (see Testing).

The Ruff hooks run uv run --locked ruff from the repository root, so local hooks and CI use the version in the root uv.lock. Each directory still uses its own lint configuration. To upgrade Ruff, run uv lock --upgrade-package ruff from the root, then run the hooks above.

Previewing Documentation

From the repository root, start the documentation site with live reload and open it in your browser:

uv run --extra docs properdocs serve --strict --open

The preview is served at http://localhost:8000/ and automatically rebuilds when files under docs/ or properdocs.yml change.

Testing Runner Changes

Most runner changes can be tested against the local Minikube stack without Pulumi or AWS. With Minikube running (see Local Minikube Setup), rebuild the runner under a new tag and submit an eval to the local API. Run these commands from hawk/:

IMAGE_TAG=my-tag
RUNNER_IMAGE_NAME=localhost:5000/runner \
  PYTHON_BASE_IMAGE=python:3.13-slim-trixie@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b \
  ../scripts/dev/build-and-push-runner-image.sh "$IMAGE_TAG"
HAWK_API_URL=http://localhost:8080 \
  hawk eval-set examples/simple.eval-set.yaml --image-tag "$IMAGE_TAG"

The real runner uses the model configured in the eval set, so configure its provider key in hawk/.env before starting the local stack.

To test against a deployed dev stack instead, push the image to that stack's ECR and submit the eval to the same stack:

# From the repository root, after `pulumi login` to the deployment's S3
# backend and authenticating AWS and Docker to the stack's ECR:
export AWS_PROFILE="<profile-for-the-target-account>"
export PULUMI_FALLBACK_TO_STATE_SECRETS_MANAGER=true
export STACK="dev-<name>"
IMAGE_TAG=my-tag
ENVIRONMENT="$(pulumi stack output env -s "$STACK")"
AWS_REGION="$(pulumi stack output region -s "$STACK")"
(
  cd hawk
  PULUMI_STACK="$STACK" \
    ENVIRONMENT="$ENVIRONMENT" AWS_REGION="$AWS_REGION" \
    ../scripts/dev/build-and-push-runner-image.sh "$IMAGE_TAG"
)
# Pin the CLI's API, Middleman, and viewer URLs to the same stack.
uv run python scripts/dev/generate-env.py "$STACK" > .env
hawk login  # use --no-browser in a headless environment
hawk eval-set hawk/examples/simple.eval-set.yaml --image-tag "$IMAGE_TAG"

Local Minikube Setup

This runs Hawk entirely locally. It uses MinIO for S3, local PostgreSQL, a local Docker registry, and Minikube for Kubernetes. This is the same setup used by E2E tests in CI.

Prerequisites

You must be inside the devcontainer, which includes minikube, Docker-in-Docker, cilium, kubectl, helm, and gvisor.

Quick Start

These commands are run from the hawk/ directory:

cp .env.example .env
../scripts/dev/start-minikube.sh

The script will:

  1. Start Minikube with gvisor, containerd, and an insecure local registry
  2. Create Kubernetes resources and install Cilium
  3. Launch services (API server, MinIO, PostgreSQL, Docker registry)
  4. Verify cluster access to the registry and configure MinIO
  5. Build and push dummy and real runner images
  6. Log in through the bundled Dex provider and run a simple eval set

Running Evals Locally

HAWK_API_URL=http://localhost:8080 hawk eval-set examples/simple.eval-set.yaml --image-tag=dummy

Run k9s to monitor the Inspect pod.

For rebuilding and selecting a real runner image after code changes, see Testing Runner Changes.

Updating Dependencies (Inspect AI / Inspect Scout)

Use the prepare-release.py script:

# Update to a specific PyPI version
scripts/ops/prepare-release.py --inspect-ai 0.3.50

# Update to a specific git commit SHA
scripts/ops/prepare-release.py --inspect-ai abc123def456

# Update Scout
scripts/ops/prepare-release.py --inspect-scout 0.2.10

The script updates pyproject.toml files, runs uv lock, creates a release branch (for PyPI versions), and publishes any npm packages if needed.

Bumping inspect-ai also moves what runner jobs get. The runner builds a fresh venv per job, and that venv is pinned to the same inspect-ai hawk itself is running — read from package metadata, which in a deployed runner is the uv sync --locked result baked into the image. Until you bump, jobs stay on the pinned version even if a newer one ships upstream.

That pin is a hard constraint, not a preference: a task or scanner package that requires a newer inspect-ai makes the venv unresolvable, and the job fails at install rather than silently upgrading. So the staleness of this pin is user-visible — if people start hitting no version of inspect-ai==<pinned>, that is the signal to bump. Users can opt out per-config via packages: (see Overriding inspect-ai or inspect-scout).

Database Migrations

See Database for migration instructions.

Pull Requests

When creating PRs, use the template at .github/pull_request_template.md. The template includes:

  • Overview and linked issue
  • Approach and alternatives considered
  • Testing & validation checklist
  • Code quality checklist
  • Before-merging checklist: signed/Verified commits and a Conventional-Commit title