Architecture

Hawk provides infrastructure for running Inspect AI evaluations in a Kubernetes environment using YAML configuration files.

Overview

flowchart TD
    CLI["<b>hawk CLI</b>"]
    Client["<b>hawk.client</b><br/><i>scripts · hawk[client]</i>"]
    API["Hawk API Server<br/><i>FastAPI on ECS Fargate</i>"]
    EKS["Kubernetes (EKS)<br/><i>Scaled by Karpenter</i>"]
    Runner["Runner Pod<br/><i>Creates virtualenv, runs inspect_ai.eval_set()</i>"]
    Sandbox["Sandbox Pod(s)<br/><i>Isolated execution · Cilium network policies</i>"]
    TB["Token Broker<br/><i>Lambda (VPC) behind ALB<br/>JWT → scoped AWS creds</i>"]
    S3[("S3<br/><i>Eval logs</i>")]
    S3Events["EventBridge<br/><i>AWS default event bus</i>"]
    HawkEvents["EventBridge<br/><i>Hawk custom bus</i>"]
    JobStatusUpdated["job_status_updated<br/><i>Lambda · tags files, emits events</i>"]
    Importer["Importer<br/><i>AWS Batch · parses .eval files</i>"]
    DB[("Aurora PostgreSQL<br/><i>Results warehouse</i>")]
    Viewer["Web Viewer<br/><i>ECS Fargate · static SPA · Browse, filter, export</i>"]
    Middleman["Middleman<br/><i>LLM Proxy · ECS Fargate</i>"]
    LLMs["LLM Providers<br/><i>OpenAI · Anthropic · Google · etc.</i>"]
    Valkey[("Valkey<br/><i>ElastiCache Serverless · optional shared cache</i>")]

    CLI --> API
    Client --> API
    API -- "Validates config & auth<br/>Creates Helm release" --> EKS
    EKS --> Runner
    EKS --> Sandbox
    Runner -- "Exchange JWT" --> TB
    TB -- "Scoped STS creds" --> Runner
    Runner -- "Writes logs<br/>(scoped creds)" --> S3
    Runner <-- "API calls" --> Middleman
    Middleman --> LLMs
    Middleman -- "Shared caches" --> Valkey
    S3 -- "Object-created event" --> S3Events
    S3Events -- "Object-created rule" --> JobStatusUpdated
    JobStatusUpdated -- "Submits Batch job" --> Importer
    Importer --> DB
    Viewer -- "Browser calls" --> API

Detailed Architecture

graph TB
    subgraph "User's Computer"
        CLI[hawk CLI]
        Client[hawk.client]
        WEB[Web viewer<br/>browser]
        IAMCLIENT[AWS CLI or SDK<br/>Identity Center role session · optional]
    end

    subgraph "Hawk API Service"
        API[FastAPI Server]
        AUTH[JWT Auth]
    end

    subgraph "Kubernetes Control Plane"
        HELM1[Helm Release 1<br/>Hawk]
        CHART1[Helm Chart 1<br/>Inspect Runner Job Template]

        HELM2[Helm Release 2<br/>inspect-k8s-sandbox]
        CHART2[Helm Chart 2<br/>Sandbox Environment Template]
    end

    subgraph "Inspect Runner Pod"
        HAWKLOCAL[hawk local]
        VENV[Virtual Environment]
        RUNNER[hawk.runner]
        INSPECT[inspect_ai.eval_set]
        K8SSANDBOX[inspect_k8s_sandbox]
    end

    subgraph "Sandbox Environment"
        POD2[Sandbox Environment Pod]
    end

    subgraph "AWS Infrastructure"
        S3[S3 Bucket<br/>Log Storage]
        S3Events[EventBridge<br/>AWS default event bus]
        HawkEvents[EventBridge<br/>Hawk custom bus]
        L1[job_status_updated<br/>Lambda]
        L2[eval_log_reader<br/>Lambda]
        L3[token_broker<br/>Lambda · VPC + ALB]
        L4[scan_importer<br/>Lambda + SQS]
        BATCH[eval_log_importer<br/>AWS Batch]
        SE[sample_editor<br/>AWS Batch]
        OL[S3 Object Lambda<br/>Access Point · optional]
        AURORA[(Aurora PostgreSQL<br/>Warehouse)]
    end

    CLI -->|Authenticated HTTP| API
    Client -->|Authenticated HTTP| API
    WEB -->|Authenticated API requests| API
    API -->|Authenticate| AUTH
    API -->|Authorized log access| S3
    CLI -. "Presigned GET<br/>(after API authorization)" .-> S3
    WEB -. "Presigned GET<br/>(after API authorization)" .-> S3
    API -->|Create Release| HELM1
    HELM1 -->|Deploy| CHART1
    CHART1 -->|Run| HAWKLOCAL
    HAWKLOCAL -->|Create venv| VENV
    VENV -->|Execute| RUNNER
    RUNNER -->|Call| INSPECT
    INSPECT -->|Invoke| K8SSANDBOX
    K8SSANDBOX -->|Create Release| HELM2
    HELM2 -->|Deploy| CHART2
    CHART2 -->|Create Pod| POD2
    INSPECT -->|Run Commands| POD2

    RUNNER -->|Get Scoped Credentials| L3
    L3 -->|Validate Permissions| S3
    INSPECT -->|Write Logs| S3
    S3 -->|Object-created event| S3Events
    S3Events -->|Object-created rule| L1
    L1 -->|Submit import job| BATCH
    L1 -->|Scan completion event| HawkEvents
    BATCH -->|Insert rows| AURORA
    HawkEvents -->|ScannerCompleted per scanner| L4
    L4 -->|Insert rows| AURORA
    S3Events -->|Sample-edit object created| SE
    SE -->|Update .eval files| S3
    IAMCLIENT -->|GetObject / HeadObject| OL
    OL -->|Check Permissions| L2
    L2 -->|Read metadata and authorized object| S3

Components

CLI (hawk)

The CLI is the usual interface for submitting evals and scans. It parses YAML, stores tokens in the keyring, and calls the Hawk API through hawk.client.HawkClient.

Python client (hawk.client)

Scripts can talk to the same Hawk API without Click or Inspect: pip install hawk[client]. Auth is token=, HAWK_ACCESS_TOKEN, or a token from hawk login. See the Python client page.

API Server

FastAPI-based REST API running on ECS Fargate. Handles JWT authentication, job orchestration via Helm, and configuration validation using Pydantic models.

Helm Chart: Inspect Runner Job Template

Each job gets its own isolated namespace. Resources created per job:

• Namespace — runner namespace, plus a separate sandbox namespace for eval sets
• Job — the Kubernetes job that runs the evaluation
• ConfigMap — stores the eval set configuration and per-job kubeconfig
• Secret — API keys, common env vars, and user-provided secrets
• ServiceAccount — with AWS IAM role annotation and RoleBinding
• CiliumNetworkPolicy — network isolation allowing egress only to sandbox namespace, kube-dns, API server, and external services

Namespace Naming

• Runner namespace: {prefix}-{job_id} (e.g., inspect-my-eval-123)
• Sandbox namespace: {runner_namespace}-s (e.g., inspect-my-eval-123-s)
• Maximum job ID: 43 characters (enforced to stay within Kubernetes' 63-char namespace limit)

Runner (hawk.runner.entrypoint)

The runner pod's entrypoint creates an isolated Python virtualenv, installs dependencies (inspect_k8s_sandbox, task/solver/model packages), then executes the evaluation. This isolation prevents dependency conflicts.

Sandbox Environments

Sandbox pods run within a StatefulSet (auto-recreated on crash) with resource constraints and limited network access, provided by inspect_k8s_sandbox.

Log Storage

Evaluation logs are written directly to S3 by Inspect AI:

• *.eval — individual evaluation result files (zip archives)
• logs.json — metadata mapping eval file paths to headers
• .buffer/ — in-progress state (SQLite databases)

Token Broker

An AWS Lambda in the VPC, sitting behind the environment's ALB at token-broker.hawk.<domain> (token-broker-<name>.hawk.<staging-domain> for a dev environment). It has no public Function URL, and its DNS name is published only in the private hosted zone, so in practice it is reached from inside the VPC, primarily by runner pods. Routing is a host-header rule on the environment's ALB listener. If that ALB is configured as internet-facing (hawk:albInternal: "false"), the endpoint is not strictly network-isolated; the security boundary is the JWT required on every request and the scoping of the credentials it issues.

It exchanges a user JWT for short-lived AWS STS credentials scoped to the specific job that requested them. Scoping is enforced two ways: a session policy (e.g. allow s3:GetObject only under evals/<job_id>/* or scans/<job_id>/*), and AWS session tags that downstream IAM policies can use for ABAC.

For eval-set runners this means the credentials can read/write only that eval-set's S3 prefix; for scan runners, a pre-allocated set of read-slots covering the eval-sets the scan operates over. This is how multi-tenant isolation is enforced at the AWS credential level, not just the application level.

Asynchronous Processing

Component	Type	Purpose
job_status_updated	Lambda	Triggered by S3 EventBridge object-created events under evals/ and scans/. Tags files by model, submits the warehouse import as a Batch job, and publishes scan-completion events to EventBridge.
eval_log_importer	AWS Batch	Submitted directly by whichever component wants an eval imported. Parses .eval files and writes rows to the Aurora warehouse.
scan_importer	Lambda + SQS	Consumes scan completion events. Writes scan results to the warehouse.
sample_editor	AWS Batch	Edits eval samples post-execution (e.g. for redaction or correction).
eval_log_reader	S3 Object Lambda	Optional IAM role-session read path. Enforces object model-group requirements for GetObject and HeadObject through Identity Store membership or the deployment's public-only policy.
token_broker	Lambda (VPC + ALB)	See Token Broker above.

eval_log_importer also runs a long-lived live-ingest consumer (a Fargate service draining an SQS queue fed by the runner's .buffer/*/manifest.json writes), gated by hawk:liveEvalImportEnabled (default off). When enabled, it pulls in-flight sample and event data for still-running eval sets straight out of the runner's S3 buffer within about a minute of each manifest sync and writes it to the warehouse flagged is_provisional=True, ahead of the terminal .eval import that supersedes it. A sample that completes mid-run is pruned from that buffer soon after; while its completed summary is still in the manifest the consumer records its completion and scores on the provisional row, and the terminal import replaces the row at the end of the eval. The consumer keeps one committed read offset per eval on the eval row, at the granularity of one sample's delta in one segment, advanced in the same transaction as the rows it covers; a crash replays at most one member through idempotent upserts. This changes some previously-safe assumptions, deliberately: total_samples/completed_samples still read 0 mid-run, because a partial log has results=None; and an in-flight sample's raw status column reads 'success' with a NULL completed_at (the buffer only carries samples inspect has not marked complete), so the CLI and viewer render is_provisional rows as "Running" rather than trusting status — is_provisional is the honest signal that a row may still change, and status cannot be, because sample.status is a stored generated column over an IMMUTABLE function, so a genuine 'running' value would require rewriting the table. Live data is best-effort; the terminal .eval import remains the correctness guarantee. It also depends on a runner setting: the S3 buffer only exists when the runner's log_shared option is enabled, so enabling the flag without that setting on produces nothing, silently.

Eval imports get two total attempts on Fargate Spot. If the final failure is a Spot interruption, EventBridge submits one on-demand attempt; other failures go to the importer DLQ. Dev environments share staging's Spot and on-demand queues.

Log Access Flow

Hawk has two end-user authorization paths for reading logs.

Built-in CLI and web viewer

1. hawk web opens the viewer, whose browser requests logs from the Hawk API's /view/logs routes. hawk transcript and related CLI commands call those authenticated API routes directly.
2. The API maps the requested path to the eval-log S3 prefix and checks the folder's model groups against the permissions in the caller's Hawk authentication context.
3. After authorization, the API either streams S3 data with credentials from its ECS task role or issues a short-lived presigned S3 URL. Clients fetch directly from S3 only on the latter paths, such as hawk download and viewer downloads; hawk transcript streams through the API.

Presigned URLs are bearer capabilities: S3 does not re-check Hawk permissions, and a holder may use a URL while it remains valid.

Optional S3 Object Lambda access point

When hawk:enableS3ObjectLambda is true, the deployment also creates an access point for separately authorized IAM Identity Center role sessions:

S3 Object Lambda availability

Since November 7, 2025, AWS makes S3 Object Lambda available only to customers already using the service and select AWS Partner Network partners. Other deployments should leave this option disabled. See AWS's availability notice.

Unless hawk:publicModelsOnly is true, reads through this path also require hawk:identityStoreId and hawk:identityStoreRegion; Pulumi does not currently validate those settings during deployment.

1. An Identity Center role session that has been granted access calls GetObject or HeadObject through the S3 Object Lambda access point.
2. S3 invokes eval_log_reader, which enforces the object's required model groups, then asks the database's code policy (public.code_policy_has_access) about the run's code. In normal mode it checks the corresponding Identity Store user's group memberships; public-models-only mode allows only objects whose model requirements are public, and still looks up the user's groups when the run's code is restricted. The code check applies in both modes and has no admin or owner exception. A missing .models.json is a 404; it answers 503 when the code policy or the Identity Store can't be read.
3. If authorized, eval_log_reader returns the object or metadata through the access point.

This path is separate from the built-in CLI and web viewer; neither routes log reads through S3 Object Lambda.

End users should not be granted direct access to the underlying S3 bucket. They should use the Hawk API, including presigned URLs it issues, or the optional S3 Object Lambda access point.

Repository Structure

infra/                  Pulumi infrastructure (Python)
  __main__.py           Entrypoint
  core/                 VPC, EKS, ALB, ECS, RDS, Route53, S3
  k8s/                  Karpenter, Cilium, Datadog agent, RBAC
  hawk/                 Hawk API + web viewer (ECS), Lambdas, Batch jobs, EventBridge
  datadog/              Monitors, dashboards (optional)
  lib/                  Shared config, naming, tagging helpers

hawk/                   Hawk application (Python + React)
  client/               Public async Hawk API client (hawk[client])
  cli/                  CLI (Click-based; uses hawk.client for API HTTP)
  api/                  API server (FastAPI)
  runner/               Kubernetes job runner
  core/                 Shared types, DB models, log importer
  www/                  Web viewer (React + TypeScript + Vite)
  services/             Lambda and Batch function source code
  examples/             Example eval and scan configs
  tests/                Unit, E2E, and smoke tests

middleman/              LLM proxy (OpenAI, Anthropic, Google Vertex)

Pulumi.example.yaml     Documented config reference