Architecture¶

Hawk provides infrastructure for running Inspect AI evaluations in a Kubernetes environment using YAML configuration files.

Overview¶

flowchart TD
    User["<b>hawk eval-set config.yaml</b>"]
    API["Hawk API Server<br/><i>FastAPI on ECS Fargate</i>"]
    EKS["Kubernetes (EKS)<br/><i>Scaled by Karpenter</i>"]
    Runner["Runner Pod<br/><i>Creates virtualenv, runs inspect_ai.eval_set()</i>"]
    Sandbox["Sandbox Pod(s)<br/><i>Isolated execution · Cilium network policies</i>"]
    TB["Token Broker<br/><i>Lambda (VPC) behind ALB<br/>JWT → scoped AWS creds</i>"]
    S3[("S3<br/><i>Eval logs</i>")]
    EU["eval_updated<br/><i>Lambda · tags files, emits events</i>"]
    Importer["Importer<br/><i>AWS Batch · parses .eval files</i>"]
    DB[("Aurora PostgreSQL<br/><i>Results warehouse</i>")]
    Viewer["Web Viewer<br/><i>CloudFront · Browse, filter, export</i>"]
    Middleman["Middleman<br/><i>LLM Proxy · ECS Fargate</i>"]
    LLMs["LLM Providers<br/><i>OpenAI · Anthropic · Google · etc.</i>"]
    Valkey[("Valkey<br/><i>ElastiCache Serverless · optional shared cache</i>")]

    User --> API
    API -- "Validates config & auth<br/>Creates Helm release" --> EKS
    EKS --> Runner
    EKS --> Sandbox
    Runner -- "Exchange JWT" --> TB
    TB -- "Scoped STS creds" --> Runner
    Runner -- "Writes logs<br/>(scoped creds)" --> S3
    Runner <-- "API calls" --> Middleman
    Middleman --> LLMs
    Middleman -- "Shared caches" --> Valkey
    S3 -- "S3 event" --> EU
    EU -- "EventBridge" --> Importer
    Importer --> DB
    Viewer --> DB

Detailed Architecture¶

graph TB
    subgraph "User's Computer"
        CLI[hawk eval-set]
    end

    subgraph "Hawk API Service"
        API[FastAPI Server]
        AUTH[JWT Auth]
    end

    subgraph "Kubernetes Control Plane"
        HELM1[Helm Release 1<br/>Hawk]
        CHART1[Helm Chart 1<br/>Inspect Runner Job Template]

        HELM2[Helm Release 2<br/>inspect-k8s-sandbox]
        CHART2[Helm Chart 2<br/>Sandbox Environment Template]
    end

    subgraph "Inspect Runner Pod"
        HAWKLOCAL[hawk local]
        VENV[Virtual Environment]
        RUNNER[hawk.runner]
        INSPECT[inspect_ai.eval_set]
        K8SSANDBOX[inspect_k8s_sandbox]
    end

    subgraph "Sandbox Environment"
        POD2[Sandbox Environment Pod]
    end

    subgraph "AWS Infrastructure"
        S3[S3 Bucket<br/>Log Storage]
        EB[EventBridge]
        L1[eval_updated<br/>Lambda]
        L2[eval_log_reader<br/>Lambda]
        L3[token_broker<br/>Lambda · VPC + ALB]
        L4[scan_importer<br/>Lambda + SQS]
        BATCH[eval_log_importer<br/>AWS Batch]
        SE[sample_editor<br/>AWS Batch]
        OL[S3 Object Lambda<br/>Access Point · optional]
        AURORA[(Aurora PostgreSQL<br/>Warehouse)]
    end

    CLI -->|HTTP Request| API
    API -->|Authenticate| AUTH
    API -->|Create Release| HELM1
    HELM1 -->|Deploy| CHART1
    CHART1 -->|Run| HAWKLOCAL
    HAWKLOCAL -->|Create venv| VENV
    VENV -->|Execute| RUNNER
    RUNNER -->|Call| INSPECT
    INSPECT -->|Invoke| K8SSANDBOX
    K8SSANDBOX -->|Create Release| HELM2
    HELM2 -->|Deploy| CHART2
    CHART2 -->|Create Pod| POD2
    INSPECT -->|Run Commands| POD2

    RUNNER -->|Get Scoped Credentials| L3
    L3 -->|Validate Permissions| S3
    INSPECT -->|Write Logs| S3
    S3 -->|Object Created Event| L1
    L1 -->|Emit completion event| EB
    EB -->|Trigger import| BATCH
    BATCH -->|Insert rows| AURORA
    EB -->|Scan completion| L4
    L4 -->|Insert rows| AURORA
    EB -->|Edit sample| SE
    SE -->|Update .eval files| S3
    CLI -->|Read Logs| OL
    OL -->|Check Permissions| L2
    L2 -->|Authorized Access| S3

Components¶

CLI (`hawk`)¶

The CLI is the primary interface for users. It handles configuration file parsing, API communication, and credential storage using keyring.

API Server¶

FastAPI-based REST API running on ECS Fargate. Handles JWT authentication, job orchestration via Helm, and configuration validation using Pydantic models.

Helm Chart: Inspect Runner Job Template¶

Each job gets its own isolated namespace. Resources created per job:

Namespace — runner namespace, plus a separate sandbox namespace for eval sets
Job — the Kubernetes job that runs the evaluation
ConfigMap — stores the eval set configuration and per-job kubeconfig
Secret — API keys, common env vars, and user-provided secrets
ServiceAccount — with AWS IAM role annotation and RoleBinding
CiliumNetworkPolicy — network isolation allowing egress only to sandbox namespace, kube-dns, API server, and external services

Namespace Naming¶

Runner namespace: {prefix}-{job_id} (e.g., inspect-my-eval-123)
Sandbox namespace: {runner_namespace}-s (e.g., inspect-my-eval-123-s)
Maximum job ID: 43 characters (enforced to stay within Kubernetes' 63-char namespace limit)

Runner (`hawk.runner.entrypoint`)¶

The runner pod's entrypoint creates an isolated Python virtualenv, installs dependencies (inspect_k8s_sandbox, task/solver/model packages), then executes the evaluation. This isolation prevents dependency conflicts.

Sandbox Environments¶

Sandbox pods run within a StatefulSet (auto-recreated on crash) with resource constraints and limited network access, provided by inspect_k8s_sandbox.

Log Storage¶

Evaluation logs are written directly to S3 by Inspect AI:

*.eval — individual evaluation result files (zip archives)
logs.json — metadata mapping eval file paths to headers
.buffer/ — in-progress state (SQLite databases)

Token Broker¶

An AWS Lambda in the VPC, sitting behind the shared ALB at token-broker.<domain>. It has no public Function URL, and its DNS name is published only in the private hosted zone, so in practice it's reached from inside the VPC (runner pods, primarily). Note that routing is a host-header rule on the shared ALB listener — when the ALB is internet-facing (hawk:albInternal: "false", the default) the endpoint is not strictly network-isolated; the security boundary is the JWT required on every request and the scoping of the credentials it issues.

It exchanges a user JWT for short-lived AWS STS credentials scoped to the specific job that requested them. Scoping is enforced two ways: a session policy (e.g. allow s3:GetObject only under evals/<job_id>/* or scans/<job_id>/*), and AWS session tags that downstream IAM policies can use for ABAC.

For eval-set runners this means the credentials can read/write only that eval-set's S3 prefix; for scan runners, a pre-allocated set of read-slots covering the eval-sets the scan operates over. This is how multi-tenant isolation is enforced at the AWS credential level, not just the application level.

Asynchronous Processing¶

Component	Type	Purpose
eval_updated	Lambda	Triggered by S3 events on `.eval` and `logs.json` creation. Tags files by model, publishes completion events to EventBridge.
eval_log_importer	AWS Batch	Consumes EventBridge completion events. Parses `.eval` files and writes rows to the Aurora warehouse.
scan_importer	Lambda + SQS	Consumes scan completion events. Writes scan results to the warehouse.
sample_editor	AWS Batch	Edits eval samples post-execution (e.g. for redaction or correction).
eval_log_reader	S3 Object Lambda	Optional. Filters S3 GetObject responses by user model-group permissions when `hawk:enableS3ObjectLambda: "true"`.
token_refresh	Lambda	Refreshes OAuth access tokens used by other Lambda functions.
token_broker	Lambda (VPC + ALB)	See Token Broker above.

Log Access Flow¶

User requests logs via hawk web or hawk transcript
Request routes through S3 Object Lambda Access Point
eval_log_reader Lambda validates user permissions against model groups
Authorized users receive the requested log data

Users should never access the underlying S3 bucket directly — always through the Object Lambda Access Point.

Repository Structure¶

infra/                  Pulumi infrastructure (Python)
  __main__.py           Entrypoint
  core/                 VPC, EKS, ALB, ECS, RDS, Route53, S3
  k8s/                  Karpenter, Cilium, Datadog agent, RBAC
  hawk/                 Hawk API (ECS), Lambdas, EventBridge, CloudFront
  datadog/              Monitors, dashboards (optional)
  lib/                  Shared config, naming, tagging helpers

hawk/                   Hawk application (Python + React)
  cli/                  CLI (Click-based)
  api/                  API server (FastAPI)
  runner/               Kubernetes job runner
  core/                 Shared types, DB models, log importer
  www/                  Web viewer (React + TypeScript + Vite)
  services/             Lambda and Batch function source code
  examples/             Example eval and scan configs
  tests/                Unit, E2E, and smoke tests

middleman/              LLM proxy (OpenAI, Anthropic, Google Vertex)

Pulumi.example.yaml     Documented config reference