Running Evaluations¶
The Eval Set Config¶
An eval set config is a YAML file that defines a grid of tasks, agents, and models to evaluate. Hawk runs every combination.
Here's a minimal example:
tasks:
- package: git+https://github.com/UKGovernmentBEIS/inspect_evals@dac86bcfdc090f78ce38160cef5d5febf0fb3670
name: inspect_evals
items:
- name: mbpp
models:
- package: openai
name: openai
items:
- name: gpt-4o-mini
limit: 1 # optional: cap the number of samples
Submit it:
Model declarations¶
Declare every model the evaluation may call in models or model_roles, including
models selected through task arguments, monitors, graders, and fallbacks. Ordinary
launches and resumes require at least one real evaluation model; none, none/none,
and scanner-only declarations do not count. Each declared model must resolve to a
Middleman access group. Registered human evaluations started with hawk human eval
start may omit models; setting human_eval options alone does not grant this exception.
Have task code use inspect_ai.model.get_model() for the model under evaluation and
get_model(role="grader", required=True) for auxiliary models declared in model_roles.
This keeps model selection and access declarations together instead of duplicating
model names in task code or task arguments.
Package declarations must also follow the code source requirements, including on resumes.
Admission priority¶
Use --priority low|medium|high to choose the priority for cluster admission and
inference requests. The default is medium for both launches and explicit resumes:
Each explicit resume uses its current option or the default, including when
--config supplies a replacement config. Automatic retries keep the job's priority.
On a deployment that enrolls this job type in Kueue, these map to numeric priorities
-100, 0, and 100. These values are Hawk's convention: their ordering matters,
and the spacing has no special scheduling effect. Within a shared queue, priority
orders waiting, enrolled work from every user and Hawk deployment in the cluster.
It grants no capacity entitlement or permission, does not preempt running work, and
does not guarantee a reservation or start time.
Kueue uses BestEffortFIFO, so a lower-priority job that fits can be admitted while
a higher-priority job waits for a resource shape that is unavailable. Coordinate
deliberate expensive jobs with other cluster users rather than treating high as a
capacity reservation.
When a deployment enables Kueue admission, it covers runners and standard-chart sandbox services together. Each new sandbox release inherits the runner Job's current priority label; a failed read or missing label warns and keeps the submission's priority. Existing sandbox releases keep their priority. See the Kueue operator guide for configuration and the accepted multi-service partial-admission limitation.
Submission guard¶
Deployments can enable an accidental-burst guard for outstanding runner Jobs. The guard is disabled by default; when enabled, its default threshold is 128 pending, queued, or running jobs for the deployment and submitting user. Jobs that completed successfully or failed do not count. New Jobs use a stable hashed subject label. Older Jobs without that label are counted conservatively through their legacy sanitized creator label in the expected deployment namespace; this can overcount if two legacy values collide, and it does not relabel those Jobs.
The guard covers both hawk eval-set config.yaml and hawk eval-set resume. A
threshold refusal returns HTTP 429 with the observed count and limit; a count failure
returns HTTP 503. Pass --force only for a purposeful large submission that you have
coordinated with other cluster users. It skips the guard and its count for that request;
authentication, configuration validation, and native admission still apply.
Because the guard's lock is local to one API process, it reduces
accidental bursts rather than enforcing an exact distributed quota.
Adding Agents¶
agents:
- package: git+https://github.com/METR/inspect-agents@metr_agents/v0.3.20#subdirectory=packages/agents
name: metr_agents
items:
- name: react
Eval Parameters¶
These fields are passed through to inspect_ai.eval_set():
| Field | Description |
|---|---|
limit |
Maximum samples to evaluate |
time_limit |
Per-sample time limit in seconds |
message_limit |
Maximum messages per sample |
epochs |
Number of evaluation epochs |
eval_set_id |
Custom ID (auto-generated if omitted) |
metadata |
Arbitrary metadata dictionary |
tags |
List of tags for organization |
log_images |
Whether to log base64-encoded images in the eval log. Omit to use inspect-ai's default (true). |
For the complete list of fields, see the Eval Set Config Reference or the Inspect AI docs.
The submission priority applies to all Middleman calls, including model roles
and online scanners. It overrides any x-middleman-priority header supplied by
the task or model configuration.
Secrets and API Keys¶
Pass environment variables to your eval runner with --secret or --secrets-file:
# From your environment
hawk eval-set config.yaml --secret MY_API_KEY
# From a file
hawk eval-set config.yaml --secrets-file .env
# Both
hawk eval-set config.yaml --secrets-file .env --secret ANOTHER_KEY
By default, Hawk routes model API calls through its managed LLM proxy (supporting OpenAI, Anthropic, and Google Vertex). To use your own API keys instead, pass the matching key as a secret, disable the proxy's token refresh, and override that provider's base URL so it points directly upstream. For OpenAI, for example:
You can also declare required secrets in your config to catch missing credentials before the job starts:
To install private packages with your own Git credentials, put Git's environment
config in a secrets file and pass it with --secrets-file. It replaces Hawk's own
Git setup for that run:
# git.env: keep it out of version control
GIT_CONFIG_COUNT=1
GIT_CONFIG_KEY_0=url.https://oauth2:<token>@gitlab.example.com/.insteadOf
GIT_CONFIG_VALUE_0=https://gitlab.example.com/
AWS Secrets Manager¶
If your deployment enables it, the runner can fetch a secret's value from AWS Secrets Manager instead of you passing it on the command line. Set type: aws-secrets-manager and the runner reads <default prefix><name> at startup, injecting the value as the environment variable name:
runner:
secrets:
- name: HF_TOKEN
type: aws-secrets-manager
description: Hugging Face token fetched by the runner at startup
With the default namespace (inspect-tasks/), that reads the secret inspect-tasks/HF_TOKEN.
To read a secret whose name differs from the environment variable — for example when several teams need their own value behind a shared variable name — add secret_name. It resolves under the same default prefix, and the task still sees HF_TOKEN:
Use arn instead when the secret lives outside the default prefix entirely; it takes a full ARN with no wildcards. secret_name and arn are mutually exclusive.
Your deployment restricts which secrets a runner may read, so a secret_name or arn resolving outside the allowed namespace is rejected before the job starts. hawk local fetches these secrets with your own AWS credentials.
Additional Packages¶
Install extra Python packages into the runner's virtualenv:
Pin git packages to a tag or commit SHA, as above. Without a ref, each run installs whatever the branch's latest commit is at that moment, so two runs of the same config can silently use different code: results aren't reproducible, and an upstream change can break your eval without anything on your side changing. Tags can also be moved; only a SHA is guaranteed never to change.
Cluster jobs ignore registry releases under a week old, except inspect-ai. A pin like some-package==1.2.3 fails to resolve while 1.2.3 is that new; pin its git tag instead, or wait.
Private GitHub repos work automatically if Hawk's GitHub token or App has access. Both git@github.com: and ssh://git@github.com/ URL formats are supported and converted to HTTPS internally. With Hawk's GitHub App, a private repository that code-access rules restrict is fetched only if your config declares it, for example in packages, including a dependency of your task.
Overriding inspect-ai¶
Eval-set jobs install the same inspect-ai that Hawk itself runs, so a job
can't silently pick up a version Hawk hasn't been tested against. Naming
inspect-ai in packages: overrides Hawk's pin:
(Nothing in an eval-set job uses inspect-scout — it only applies to scans, see
below — so naming it here just installs an unused package. It doesn't override a
pin, because there is no inspect-scout pin on this path.)
For scans: the same override mechanism also covers inspect-scout, since
scan jobs install and pin both packages. Naming either in a scan config's
packages: drops that package's extra and its pin. Scout's exact version
doesn't come from detected install metadata the way inspect-ai's does —
the runner image never installs inspect-scout directly, so there's nothing to
introspect there — it comes from the exact pin Hawk's own pyproject.toml
declares. The outcome for a scan author is the same either way: naming it in
packages: overrides it.
The two overrides are independent — overriding inspect-scout leaves
inspect-ai pinned. Scout only declares a lower bound on inspect-ai, so
releasing both would float inspect-ai to whatever is newest rather than to a
pairing anyone has tested. If the pinned inspect-ai doesn't satisfy the Scout
you asked for, resolution fails and you should name a compatible inspect-ai
too:
See Running Scans for the scan packages: field itself.
The override is matched by package name, so the entry has to name the package. A bare URL — the form shown under Additional Packages — won't be recognised, because it names no package until it is built:
packages:
# Recognised: Hawk drops its inspect-ai pin.
- inspect-ai @ git+https://github.com/my-org/inspect_ai_fork.git@my-fix
# Not recognised: Hawk can't tell what this provides, so it keeps its pin
# and the job fails with "no version of inspect-ai==<pinned version>".
- git+https://github.com/my-org/inspect_ai_fork.git@my-fix
The same applies to any dependency that demands a newer inspect-ai than the
pinned one — a task or scanner package declaring inspect-ai>=<something newer>
makes the venv unresolvable and the job fails at install rather than quietly
upgrading. Name a compatible inspect-ai in packages: to take control.
[Experimental] Custom Runner Images¶
You can use your own Docker image for the runner instead of the default:
Or via the CLI: hawk eval-set config.yaml --image <image-uri>
Images must have an explicit tag (:v1) or digest (@sha256:...). Tagless and :latest are rejected.
The image must be multi-arch or match the deployment's hawk:cpuArchitecture
setting. A custom amd64-only runner requires an amd64 deployment.
Each Hawk deployment includes a custom-runners ECR repo with immutable tags. Get its URL with pulumi stack output custom_runners_ecr_url. Public images from any registry also work.
Look at the runner build stage in hawk/Dockerfile to see what a valid image looks like.
Sandbox Networking and Hardening¶
Sandbox network access is declared in your task's compose file or Kubernetes
values.yaml, not in the eval-set config. Sandboxes are deny-by-default: no
internet unless the task opts in, and network_mode: none isolates a service
from its siblings as well.
See Sandbox Network Isolation for the full rules.
For evals that run code you expect to be hostile (cyber and offensive-security tasks,
misuse probes), set isolation: strict and see
Hardening High-Risk Evals.
Sandbox concurrency¶
The runner sizes the k8s_sandbox concurrency knobs from its own resources at startup: INSPECT_MAX_POD_OPS (the exec thread pool, about 1.25 per sandbox in the resolved max_sandboxes, between 64 and 1024) and INSPECT_MAX_HELM_INSTALL / INSPECT_MAX_HELM_UNINSTALL (3 per CPU of the runner's cgroup limit, between 8 and 64). Anything you set in runner.environment wins over the derived value:
Sandbox architecture¶
CPU sandbox services run on the deployment's default architecture
(hawk:evalTaskArchitecture, amd64 unless the operator changed it) unless the
task pins one per service. In a compose file, use the standard platform key,
spelled exactly linux/amd64 or linux/arm64 — any other value is rejected
before the eval starts:
services:
default:
image: ghcr.io/example/agent-env:1.4 # must publish a linux/arm64 variant
platform: linux/arm64
scorer:
image: python:3.12 # no platform: deployment default
Hawk adds the Graviton pool's taint toleration for an arm64 service, and
platform: linux/amd64 keeps a service on x86 even where the deployment default
is arm64. Hawk cannot check that an image exists for the chosen architecture:
the service's image and any init containers must publish a matching variant, or
the pod fails to start. A multi-arch image needs no declaration. GPU services
always run on amd64, so platform: linux/arm64 on a service that requests
nvidia.com/gpu is rejected. Native Kubernetes values.yaml tasks express the
same thing with nodeSelector: {kubernetes.io/arch: arm64}; the x-k8s
extension does not accept nodeSelector.
A compose file whose only service is not named default also gets the chart's
implicit default pod, which follows the deployment default, so name a lone
service default (or mark it x-default: true) for the pin to apply to the pod
the agent uses.
platform also changes local runs
Docker Compose honours the same key, so platform: linux/amd64 runs under
emulation on an Apple Silicon laptop. Under hawk local with a
("k8s", "compose.yaml") sandbox the key is rejected outright, because
sandbox patching only runs when HAWK_RUNNER_PATCH_SANDBOX is set.
GPU Tasks¶
GPU node pools are tainted (nvidia.com/gpu: NoSchedule) so that only workloads
that need a GPU land on them. Any sandbox service that requests nvidia.com/gpu
in its resource limits or requests automatically gets a toleration for that
taint injected — task authors don't need to declare one. If a service already
declares its own toleration that tolerates the taint (for example operator:
Exists on the nvidia.com/gpu key), Hawk leaves the service's tolerations
untouched. Operators can change what gets injected with the
hawk:gpuNodeTolerations stack config option.
Container resource limits¶
Agents that size work from nproc, lscpu, free, or /proc/meminfo see the node, which is usually larger than the sandbox cgroup.
Hawk injects the limits that actually bind into every patched k8s sandbox service as environment variables:
CONTAINER_CPU_LIMIT(Kubernetes CPU quantity, e.g.500mor2)CONTAINER_MEMORY_LIMIT(e.g.2Gi)CONTAINER_EPHEMERAL_STORAGE_LIMIT(only when the pod spec sets one)CONTAINER_GPU_LIMIT(only when the pod requests a GPU)
If the service omits resources, these match the inspect_k8s_sandbox chart defaults (500m CPU / 2Gi memory).
A task-supplied CONTAINER_* variable is left alone.
The same numbers are available as an Inspect tool, container_specs, that reads the live cgroup inside the sandbox.
Hawk does not attach that tool, so existing evals keep their tool set.
Task authors who want it add it themselves:
from inspect_ai.solver import generate, use_tools
from inspect_ai.tool import bash
from hawk.runner.container_specs import container_specs
solver = [use_tools(container_specs(), bash()), generate()]
Monitoring¶
Logs¶
hawk logs # last 100 log lines for current job
hawk logs -f # follow logs in real-time
hawk logs -n 50 # last 50 lines
hawk logs JOB_ID -f # follow a specific job
Status¶
hawk status # JSON report: pod state, logs, metrics, performance statistics
hawk status --hours 48 # include 48 hours of log data
The performance block (queue wait, working vs. waiting sample time, model-call
and retry/error counts) is aggregated from the warehouse.
Note
Samples reach the warehouse only once their eval log is imported, so the
performance statistics cover finished samples — a mid-run report excludes
the samples still in flight. Use
hawk watch for live per-sample progress; it
reads the runner's live sample buffer instead. For scan jobs performance
is null.
Web Viewer¶
hawk web # open current eval set in browser
hawk web EVAL_SET_ID # open a specific eval set
hawk view-sample UUID # open a specific sample
Listing and Inspecting Results¶
hawk list eval-sets # list all eval sets
hawk list evals [EVAL_SET_ID] # list evals in an eval set
hawk list samples [EVAL_SET_ID] # list samples
hawk transcript UUID # download a sample transcript (markdown)
hawk transcript UUID --raw # download as raw JSON
hawk transcripts [EVAL_SET_ID] # download all transcripts
Running Locally¶
Run evals on your own machine instead of the cluster. Useful for debugging. The commands below use the repository's example file, so run them from the root of a cloned Hawk repository. Otherwise, substitute the path to your own config.
--priority defaults to medium and also works with --direct. It sets the
inference priority when requests are routed through Middleman.
This creates a fresh virtualenv in a temp directory, installs dependencies, and runs the evaluation the same way the cluster would.
Providing model API keys¶
Unlike cluster runs, hawk local does not route model calls through Hawk's managed LLM proxy (Middleman). Inspect calls each provider's API directly, so you must supply the provider API keys yourself — otherwise the run fails with authentication errors from the provider.
Set the environment variables for the providers your models use, for example:
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
hawk local eval-set hawk/examples/simple.eval-set.yaml
Or keep them in a file and load it with --secrets-file:
You can also forward individual variables from your current shell with --secret NAME (see Secrets and API Keys above). Generate keys from your provider's dashboard (e.g. platform.openai.com, console.anthropic.com).
Note
To route through a managed gateway instead of using your own keys, see Using an AI Gateway below.
Debugging with --direct¶
Use --direct to skip the virtualenv and run in your current Python environment:
This lets you set breakpoints in your IDE and debug from the start. Note that --direct installs dependencies into your current environment via uv pip install, but model-provider packages (openai, anthropic, etc.) must already be present in the environment hawk was installed into. If they're missing, add them when installing hawk:
Note
Tasks from inspect_evals require openai even with a different model provider — inspect_evals imports it at load time, and missing it makes the task registry silently fail with LookupError.
Using an AI Gateway¶
Route model calls through a managed AI gateway:
export HAWK_AI_GATEWAY_URL=https://your-gateway.example.com
hawk login
hawk local eval-set hawk/examples/simple.eval-set.yaml
Sample Editing¶
Batch edit sample scores or invalidate samples:
See Sample Editing.
Importing Eval Files¶
Upload locally-produced .eval files into Hawk's warehouse so they appear alongside natively-run eval sets:
hawk import path/to/file.eval # single file
hawk import path/to/dir/ # directory of .eval files
hawk import path/to/dir/ --name my-import # friendly name in the eval_set_id
Both hawk import and hawk scan import show detected and supplied code sources
before uploading. Confirm that the list covers all files, or cancel and rerun
with --code-sources sources.json to add missing sources. The file is a JSON array
of canonical identities, for example ["github.com/example/tasks", "inspect-ai"].
If nothing is identified, Hawk asks whether to import with unknown sources.
Both prompts default to no.
Scripts must use --yes (-y) to skip review:
Use [] in the file for no additional sources beyond log origins. --yes without
a file records unknown alongside any detected origins. Matching rules determine
code access; sources without a rule, including unknown, are public subject to
model permissions. See import source policy
for access checks, limits, and server compatibility.
Imported eval sets get IDs prefixed imported- and have metadata.imported = true set. They appear in the warehouse, viewer, and hawk download identically to natively-run eval sets.
Importing Scans¶
Upload one or more locally-produced Scout scans into the warehouse so their scanner results appear alongside natively-run scans:
hawk scan import path/to/scan_id=.../ # a single scan results directory
hawk scan import path/to/run/ # a folder of scan directories
hawk scan import path/to/run/ --name my-import
PATH is either a single Scout scan results directory (the scan_id=... directory Scout writes) or a folder containing several such directories. Each scan directory must contain a _scan.json spec and at least one per-scanner .parquet file (_summary.json is uploaded too if present). All the scans in one import land under a single fresh, imported--prefixed scan run, each with a freshly-generated scan id, so they never collide with existing warehouse scans. The scanned transcripts must already be in the warehouse: Hawk derives the models that gate access (the scanner models, including models the scanners' recorded usage shows they called, plus the models of the scanned transcripts' source eval sets, matching what a native hawk scan run over those eval sets would require) and refuses to import if any scanned transcript can't be resolved, or if you lack permission for those models.
Every model a scanner actually called must be registered with Middleman, including models it called without naming them in the scan config: finalize reads them from each parquet's scan_model_usage column and from _summary.json. A scan whose scanner used an unregistered model (for example a local model, or one called with your own provider key) is rejected with an "Unknown models in scan" error that lists the usage-only models. Editing the scan spec cannot remove them; register the model with Middleman, or re-run the scan without it.
Human-in-the-Loop Approvals¶
For sensitive workloads you can require a human decision before the agent's tool calls execute. Two config pieces work together:
approval— a chain of Inspect approvers, matched top-down against each tool call. Approvers are resolved by registry name: Inspect's built-ins (human,auto) work out of the box, and packages listed inpackagescan register their own.acp_server— a loopback TCP port (1–65535) that starts Inspect's ACP server inside the eval, so ahumanapprover parks the tool call until someone attaches and answers, instead of the eval crashing or hanging on stdin.
A minimal end-to-end test config (a one-sample game where every tool call needs approval):
acp_server: 4444
approval_timeout_minutes: 60
approval:
approvers:
- name: human
tools: ["*"]
tasks:
- package: inspect-test-utils==1.5.0
name: inspect_test_utils
items:
- name: guess_number_keep_guessing
args: { sample_count: 1, local: true, target: "3" }
models:
- package: openai
name: openai
items:
- name: gpt-4o-mini
limit: 1
The flow:
hawk eval-set approval-test.yaml # submit; the first tool call parks
hawk watch # pending_interactions goes to 1 (status stays "running")
hawk acp # bridge through the relay, open the `inspect acp` TUI
hawk acp defaults to the last job you launched from this machine; pass an eval-set id to answer a different run. It shells out to inspect acp, so you need an inspect on your PATH from an environment that has ACP support — the inspect-ai that ships with hawk[cli] does. If that binary is missing or broken, hawk acp says so and you can bridge without launching a client: hawk acp --no-launch, then point your own inspect acp --server <address> at the printed address.
In the TUI, pick the session and answer each request: approve executes the call, reject refuses just that call (the agent continues and may try something else), and terminate ends the sample gracefully — it is scored with the work done so far and recorded with a SampleLimitEvent, not an error.
Chain semantics: each tool call is handled by the first entry whose tools pattern matches; an approver that returns escalate hands the call to the next matching entry, and if every approver escalates the call is rejected. Narrow the tools patterns (they are prefix matches, e.g. "bash*") so humans only see genuinely risky calls — that is the safest way to cut approval volume, because a call that matches no policy never needed a decision in the first place.
An auto-approving approver is a noise filter, not a safety boundary
Any approver that returns approve decides that a tool call runs with nobody having seen it. Judging a shell command safe by inspecting it is unreliable: an allowlist keyed on program names has been shown to pass commands that write and then execute a file, and narrowing the character filter does not fix it. Put human (or a model-based monitor) where the safety decision actually matters, and treat anything in front of it as a convenience that only ever handles calls you would have approved anyway.
Local runs honor both settings too: hawk local eval-set starts the ACP server on the configured loopback port on your own machine, so an ACP client connects to it directly and no relay is involved.
approval_timeout_minutes bounds how long a parked approval waits before it is automatically denied — by default it is null, and a parked approval waits until someone answers it, unless another limit such as a configured time_limit (which keeps counting while the sample is parked) ends the sample first. It requires acp_server. A timed-out approval rejects that one tool call; the sample keeps running.
hawk acp requires the deployment's attach relay (relayEnabled, on by default). While approvals are parked the sample holds its sandbox, so watch pending_interactions, and answer parked approvals within hawk:expectedLongestRunDays (30 days by default): that is as long as Hawk ensures a run's credentials last, so a sample resumed after it may fail.
Budget for the volume: a model can emit several tool calls in one turn, and each one parks separately — a single turn has been observed producing ten simultaneous approvals. Narrow tools patterns accordingly, and expect a burst rather than a steady trickle.
Running Human Evaluations¶
For evaluations driven by a human instead of an LLM agent, Hawk provisions a sandbox and exposes it via SSH through the shared jumphost. config.yaml can be any eval-set YAML — the server swaps in the configured human-agent solver, clamps epochs=1 / limit=1, and defaults runner.cleanup to false so the sandbox sticks around between SSH sessions. To set args on the installed agent (e.g., user, record_session), add a human_eval.agent_args block to your config; it's shallow-merged onto the agent. Pass --no-rewrite to keep your config's agents/solvers block untouched (the SSH key is still injected on every solver). The sandbox's network comes from the task's sandbox definition and is deny-by-default, so a baseline has no internet unless the task grants it; see the human guide.
# 1. Register the human and their SSH public key (one-time)
hawk human register --name jane --ssh-key "ssh-ed25519 AAAA..."
# 2. Start the eval — provisions a sandbox and registers jane's key on the jumphost
hawk human eval start config.yaml --human jane
# 3. Get a ready-to-paste SSH command for the sandbox
hawk human eval ssh-command # uses the last-started eval-set
# — or —
hawk human eval ssh-command <eval-set-id>
ssh-command polls the eval logs for the agent's connection line, confirms whether the sandbox with that logged endpoint is live via the Kubernetes API (refusing outright if that sandbox is confirmed gone or not ready, warning but still printing if that status can't be confirmed), and prints ssh -J ssh-user@<jumphost> <user>@<sandbox-ip> -p <port>. It retains the endpoint from the log rather than replacing it with another sandbox's address. Load your private key into ssh-agent first (ssh-add /path/to/key) — the -J ProxyJump uses the agent for both the jumphost and sandbox hops. Use --timeout SECONDS to bound how long it waits for the sandbox to come up (default 600). See the human-eval guide for the full liveness-check behavior and --skip-verify (which skips only the SSH connectivity probe).
After hawk delete <eval-set-id> (or eval completion), the SSH key is removed from the jumphost and ssh-command refuses subsequent invocations.
Stopping and Deleting¶
hawk stop tells the running eval to finish gracefully: active samples are scored with whatever work they've done so far, results are written to S3, and the job exits on its own. Use this to end an eval early but keep the partial results. It also ends a runner kept alive after completion by runner.cleanup: false (e.g. human baselines).
hawk delete kills the job immediately and tears down its Kubernetes resources. Use this when you don't care about partial results and just want the job gone. Logs already written to S3 are kept. It takes either an eval-set ID or a scan-run ID, and confirms the target before tearing anything down. Scripts and other non-interactive callers must pass -y; without a terminal to prompt at, the command refuses rather than reading an answer from redirected stdin. -y also requires the ID to be given explicitly — with no prompt there is nothing to show which job the default resolved to, and that default moves whenever any command is given a job ID, including read-only ones like hawk logs.
hawk stop # gracefully stop current eval set (score partial work)
hawk stop --error # mark samples as errors (will retry if retries are configured)
hawk stop --sample UUID # stop a single sample
hawk delete # tear down the current job's Kubernetes resources (logs are kept)
hawk delete JOB_ID # delete a specific eval set's or scan run's resources
hawk delete JOB_ID -y # skip the confirmation prompt (required in scripts;
# -y always needs the ID spelled out)