Skip to content

Python client

Scripts can call the Hawk HTTP API without installing the CLI (no Click, no Inspect). The page below is generated from HawkClient docstrings at docs-build time.

Install

After a release that includes the client:

pip install "hawk[client]"
# or: uv add "hawk[client]"

From a checkout, without publishing:

pip install -e "./hawk[client]"

Task authors can read declared input identities and locations through hawk.runtime.transcript_sources(). This runner helper is separate from the HTTP client and needs no client extra.

Auth

HawkClient never opens a browser. Pass token=, or set HAWK_ACCESS_TOKEN, or reuse a token stored by hawk login on this machine. Pass api_url=, or set HAWK_API_URL in the environment, a project .env, or ~/.config/hawk-cli/env. Supplied tokens (including token callables) use this same URL resolution and do not read the keyring. With HAWK_ACCESS_TOKEN set, the stored login is never read: the environment is the whole token store (the hawk CLI also reads HAWK_REFRESH_TOKEN beside it; HawkClient does not). For saved credentials, the selected deployment supplies its own login, and each request checks that the API recipient was approved during hawk login.

Supplied tokens go wherever the API URL points

Tokens you supply (token=, a token callable, or HAWK_ACCESS_TOKEN) are not checked against a saved login: the client sends them to whatever API URL it resolves, including one from a project .env. The CLI refuses this combination; the SDK accepts it. If you export a token, set HAWK_API_URL beside it or pass api_url=, and don't run SDK scripts from untrusted checkouts.

Unrelated service settings do not invalidate an API login. After upgrading from client-ID-scoped storage, run hawk login once per deployment. Dotenv values are literal; ${VAR} interpolation is disabled. Authenticated API requests do not follow redirects; configure the final API URL. The client does not read or refresh a stored hawk login refresh token. Pass refresh_token= when submitting jobs that use user credentials so the runner can refresh on the cluster. With credential_mode="restricted", submission and resume requests omit the refresh token even if supplied. With user or omitted mode, they forward it; an omitted mode lets the server choose. The access token still authenticates every submission.

Dict-typed responses (get_usage, get_usage_history, and listing TypedDicts) are the server JSON. Fields may be added without a client major bump.

import asyncio
from hawk.client import HawkClient

async def main() -> None:
    async with HawkClient() as client:
        jobs = await client.get_jobs(mine=True, limit=5)
        print(jobs)

asyncio.run(main())

On a machine with no prior hawk login, you can pass the URL and token explicitly:

async with HawkClient(api_url="https://api.example.com", token="...") as client:
    ...

Submit an eval-set

Pass YAML as a dict. The server validates it. You do not need Inspect installed.

import asyncio
import pathlib

import ruamel.yaml
from hawk.client import HawkClient

async def main() -> None:
    yaml = ruamel.yaml.YAML(typ="safe")
    config = yaml.load(pathlib.Path("eval-set.yaml").read_text())
    async with HawkClient() as client:
        created = await client.create_eval_set(config, priority="high")
        print(created["eval_set_id"], *created["warnings"], sep="\n")

asyncio.run(main())

Optional local validation uses EvalSetConfig from hawk.core.types.evals (still no Inspect for typical configs). create_scan accepts a mapping the same way. create_scan_detailed and resume_scan_detailed return the same warnings list as part of the response. The warnings name package sources that can change between runs; the launch has already started when they arrive.

create_eval_set, create_scan, and resume_scan accept priority="low", "medium", or "high" for admission and inference. Evals default to medium; scans default to low. Each explicit resume uses its current argument or that default. Priority is a client argument, separate from the eval/scan YAML config.

Read a run's files

list_files yields every file in an eval set's folder (or a scan's, with kind="scan"), and get_file_download_urls presigns them. With recursive=False it yields only the files directly under prefix (by default the top of the folder, where the logs are) without walking artifacts/. list_level returns that level's files and its subfolders, each ending in /, from one listing: with prefix="artifacts/", one subfolder per sample that has artifacts. A URL is valid for about 15 minutes and answers GET, including HTTP Range requests, so a script can read the start of a large file without downloading all of it. A reader that runs longer, or one that sends HEAD first, needs new URLs or a downloaded copy:

import asyncio

import aiohttp
import yarl
from hawk.client import HawkClient

async def main() -> None:
    async with HawkClient() as client, aiohttp.ClientSession() as http:
        files = [f async for f in client.list_files("my-eval-set", recursive=False)]
        evals = [f for f in files if f["path"].endswith(".eval")]
        # Agents write these names, so print them escaped.
        print(repr(max(evals, key=lambda f: f["size_bytes"])["path"]))
        _, folders = await client.list_level("my-eval-set", prefix="artifacts/")
        for folder in folders:
            print(repr(folder))  # 'artifacts/<sample-uuid>/'
        async for _path, url in client.get_file_download_urls(
            "my-eval-set", ["logs.json"]
        ):
            # As signed: aiohttp would otherwise re-quote characters in the key.
            signed = yarl.URL(url, encoded=True)
            async with http.get(signed, headers={"Range": "bytes=0-65535"}) as response:
                if not response.ok:  # aiohttp's own error would print the signed URL
                    raise SystemExit(f"logs.json: HTTP {response.status}")
                print(len(await response.read()))

asyncio.run(main())

Scans with scanner models

With scanner_models (see Model scanners over everything you can read), one request launches several scans. create_scan returns the first launched one's id, and raises HawkClientError naming each scan's error if none launched; others may still have failed, so use create_scan_detailed to check each outcome. create_scan_detailed returns the whole response as a ScanCreated: siblings holds one ScanSibling per scan (scan_run_id, scanner_model, evals, transcripts, outcome, and for a failed scan error and resumable), next to excluded_evals, not_imported_eval_sets and forbidden_eval_sets. Both types are exported from hawk.client.

A refusal raises HawkAPIError, whose problem holds the parsed Problem JSON. When some evals can't be scanned, problem["code"] is "unscannable_evals" and problem["unscannable"] lists every one (eval_set_id, file unless it is a whole eval set, reason, models).

Delete a job

await client.delete_job(job_id) waits for asynchronous restricted cleanup by retrying HTTP 503 responses. It honors Retry-After (seconds or an HTTP date), using five seconds when the header is absent or invalid. Requests and waits share the client's timeout budget, which defaults to 300 seconds. Other errors are reported immediately; only a 404 tries the scan endpoint after the eval endpoint. The hawk delete CLI command uses the same retry behavior.

API

HawkClient

HawkClient(
    *,
    api_url: str | None = None,
    token: str | Callable[[], str] | None = None,
    timeout: float = _DEFAULT_TIMEOUT_SECONDS,
)

Async HTTP client for the Hawk API.

Use as async with HawkClient(...) as client:. Requests without entering the context manager raise HawkClientError.

Pass api_url and token, or omit them to read HAWK_API_URL and reuse a token from HAWK_ACCESS_TOKEN / hawk login. With HAWK_PROFILE set and no token, requests use that restricted profile's token instead, never the full login's. Does not prompt for login or refresh a stored login token; the CLI owns that.

Submission and resume methods omit refresh_token when credential_mode="restricted". User or omitted mode forwards it for runner credential renewal; an omitted mode lets the server choose.

Open a client.

Token and URL are resolved when entering async with, not here.

Parameters:

Name Type Description Default
api_url str | None

Hawk API base URL. Defaults to HAWK_API_URL from the environment or dotenv configuration.

None
token str | Callable[[], str] | None

Bearer token, or a callable returning one, called for every request (for tokens that rotate) in a worker thread (asyncio.to_thread), possibly several at once, so it must be thread-safe and must not touch loop-bound state. Defaults to the HAWK_PROFILE profile's token, else HAWK_ACCESS_TOKEN or a hawk login token. Missing either raises HawkClientError on async with, not in __init__. Supplied tokens use the same API URL resolution as saved tokens, including dotenv configuration, without reading the keyring. The caller owns destination configuration for supplied tokens.

None
timeout float

aiohttp total timeout in seconds for ordinary requests (default 300). Downloads and SSE streams use their own timeouts.

_DEFAULT_TIMEOUT_SECONDS

api_url property

api_url: str

Resolved API base URL, without a trailing slash.

aclose async

aclose() -> None

Close the underlying HTTP session.

create_eval_set async

create_eval_set(
    eval_set_config: EvalSetConfig
    | Mapping[str, Any]
    | None = None,
    *,
    eval_set_id: str | None = None,
    image: str | None = None,
    image_tag: str | None = None,
    secrets: dict[str, str] | None = None,
    log_dir_allow_dirty: bool = False,
    refresh_token: str | None = None,
    allow_sensitive_cross_lab_scan: bool = False,
    force: bool = False,
    credential_mode: CredentialMode | None = None,
    priority: JobPriority = DEFAULT_EVAL_PRIORITY,
) -> EvalSetCreated

Create an eval-set, or resume one by passing eval_set_id.

Accepts an EvalSetConfig or a config mapping. Pass None with an existing id to resume the saved config. Priority defaults to medium for every submission, including resumes.

allow_sensitive_cross_lab_scan permits private transcript inputs to be sent to another provider when the deployment operator allows it. It applies only to this submission and must be re-provided on resume.

Returns:

Type Description
EvalSetCreated

The eval-set id, plus any advisory warnings the server raised

EvalSetCreated

about the launch (empty against a server that predates them).

create_scan async

create_scan(
    scan_config: ScanConfig | Mapping[str, Any],
    *,
    image: str | None = None,
    image_tag: str | None = None,
    secrets: dict[str, str] | None = None,
    refresh_token: str | None = None,
    allow_sensitive_cross_lab_scan: bool = False,
    force: bool = False,
    credential_mode: CredentialMode | None = None,
    priority: JobPriority = DEFAULT_SCAN_PRIORITY,
) -> str

Create a Scout scan from a ScanConfig or mapping.

Priority defaults to low. Returns its scan run id; with scanner_models, the first launched sibling scan's, raising HawkClientError if none launched (create_scan_detailed lists them all).

create_scan_detailed async

create_scan_detailed(
    scan_config: ScanConfig | Mapping[str, Any],
    *,
    image: str | None = None,
    image_tag: str | None = None,
    secrets: dict[str, str] | None = None,
    refresh_token: str | None = None,
    allow_sensitive_cross_lab_scan: bool = False,
    force: bool = False,
    credential_mode: CredentialMode | None = None,
    priority: JobPriority = DEFAULT_SCAN_PRIORITY,
) -> ScanCreated

As create_scan, returning the whole response: with scanner_models, the sibling scans launched and the evals and eval sets left out.

upload_selection async

upload_selection(path: Path) -> str

Upload a file of sample uuids (one per line) and return its selection id.

resume_scan async

resume_scan(
    scan_run_id: str,
    *,
    image: str | None = None,
    image_tag: str | None = None,
    secrets: dict[str, str] | None = None,
    refresh_token: str | None = None,
    allow_sensitive_cross_lab_scan: bool = False,
    force: bool = False,
    credential_mode: CredentialMode | None = None,
    priority: JobPriority = DEFAULT_SCAN_PRIORITY,
    shards: int | None = None,
) -> str

Resume from the last checkpoint with this submission's priority.

Priority defaults to low, regardless of the original run.

resume_scan_detailed async

resume_scan_detailed(
    scan_run_id: str,
    *,
    image: str | None = None,
    image_tag: str | None = None,
    secrets: dict[str, str] | None = None,
    refresh_token: str | None = None,
    allow_sensitive_cross_lab_scan: bool = False,
    force: bool = False,
    credential_mode: CredentialMode | None = None,
    priority: JobPriority = DEFAULT_SCAN_PRIORITY,
    shards: int | None = None,
) -> ScanCreated

As resume_scan, returning the whole response with its warnings.

get_jobs async

get_jobs(
    *,
    mine: bool = True,
    limit: int = 10,
    status: JobStatus | None = None,
    waiting: bool = False,
    job_ids: Sequence[str] | None = None,
) -> list[JobListItem]

List jobs (eval-sets and scans).

Parameters:

Name Type Description Default
mine bool

If true, only jobs created by the token's sub claim.

True
limit int

Max rows. /meta/jobs serves at most _JOBS_PAGE_SIZE rows per page, so a larger limit walks pages.

10
status JobStatus | None

Optional status filter.

None
waiting bool

If true, only jobs with pending human interactions.

False
job_ids Sequence[str] | None

Only these jobs (at most 100 per request).

None

get_eval_sets async

get_eval_sets(
    *, limit: int | None = None, search: str | None = None
) -> list[EvalSetInfo]

List eval-sets.

get_evals async

get_evals(
    eval_set_id: str, *, page: int = 1, limit: int = 100
) -> list[EvalInfo]

List evals in an eval-set.

get_samples async

get_samples(
    eval_set_id: str,
    *,
    search: str | None = None,
    page: int = 1,
    limit: int = 50,
) -> list[SampleListItem]

List one page of samples in an eval-set.

get_all_samples_for_eval_set async

get_all_samples_for_eval_set(
    eval_set_id: str, *, limit: int | None = None
) -> list[SampleListItem]

Walk sample pages until exhausted, or until limit rows.

search_transcripts async

search_transcripts(
    query: str,
    *,
    eval_set_id: str | None = None,
    sort: Literal["rank", "recency"] = "rank",
    limit: int = 50,
    cursor: str | None = None,
    diversify: bool = True,
    time_limited: bool = False,
) -> SearchResultPage

One page of full-text transcript search (GET /meta/search).

Pass the returned next_cursor back as cursor, with the same query, scope, sort and diversify, until it is None. Cursors expire after one hour. diversify=False lifts the viewer's per-transcript hit cap. time_limited=True answers within seconds instead of failing when the index is slow: such a page can be an unranked, newest-first scan, often empty but with a cursor, and its cursor needs time_limited=True too. The scan skips attachment text and ignores diversify.

stream_search_csv async

stream_search_csv(
    query: str, *, eval_set_id: str | None = None
) -> AsyncIterator[bytes]

Stream every full-text match as CSV (GET /meta/search.csv).

One request; the server walks the pages and flushes each as rows. If it stops early the last line is #resume-cursor=<cursor>; resume with search_transcripts(..., sort="recency", diversify=False, cursor=...). A query too broad to rank exports only its 10,000 newest matches and then ends with #truncated=too-broad,limit=10000, which nothing resumes. If such an export fails on a later page it ends with #truncated=error instead: no cursor resumes it and the rows are incomplete. Callers must handle both #truncated= lines (the request opts in to them) and treat any reason but too-broad as a failed export.

get_me async

get_me() -> dict[str, Any]

The caller's identity as the server resolves it.

Returns the GET /auth/me JSON: sub, email (the claim named by the deployment's email field, or None), exp, is_admin, model_groups (each with read, write and execute flags) and code_groups, as this deployment enforces them.

create_delegated_session async

create_delegated_session(
    request: DelegatedSessionCreateRequest,
) -> DelegatedSessionToken

Start a delegated session for a restricted profile and get its first token.

Needs a full OIDC login; delegated tokens are refused.

fetch_delegated_session_token async

fetch_delegated_session_token(
    session_id: str,
) -> DelegatedSessionToken

Mint a new token for one of the caller's delegated sessions.

Raises HawkAPIError with 404 (unknown session), 410 (session ended) or 403 (nothing left to grant).

list_delegated_sessions async

list_delegated_sessions() -> list[DelegatedSessionInfo]

The caller's active delegated sessions.

delete_delegated_session async

delete_delegated_session(session_id: str) -> None

End one of the caller's delegated sessions; its tokens stop being renewed.

get_usage async

get_usage(group_by: str = 'user') -> list[dict[str, Any]]

Current token usage, grouped by provider and model.

group_by selects the split: "user" (the default, filling each model's users), "team" (teams), "job" (jobs) or "task" (tasks). The parameter is only sent when it differs from the default, so servers predating it see the same request as before.

Returns the server JSON list. Fields may be added without a client major bump.

get_usage_history async

get_usage_history(
    start: int,
    end: int,
    bin_seconds: int,
    group_by: str = "user",
) -> list[dict[str, Any]]

Binned token usage between Unix timestamps start and end.

group_by selects the split: "user" (the default, filling each bin's users), "team" (teams), "job" (jobs) or "task" (tasks). The parameter is only sent when it differs from the default, so servers predating it see the same request as before.

Returns the server JSON list. Fields may be added without a client major bump.

get_log_files async

get_log_files(eval_set_id: str) -> list[LogFileInfo]

List .eval log files for an eval-set.

get_log_headers async

get_log_headers(file_names: list[str]) -> list[EvalHeader]

Fetch partial eval-log headers for the given file names.

get_download_url async

get_download_url(log_path: str) -> tuple[str, str]

Presign one log file.

Returns:

Type Description
tuple[str, str]

(url, filename). Fetch the URL yourself; this is not an S3 client.

get_download_urls async

get_download_urls(
    log_paths: list[str],
) -> AsyncIterator[tuple[str, str]]

Yield presigned S3 URLs for log files as each batch completes.

Batches of BATCH_DOWNLOAD_URLS_LIMIT paths are posted concurrently.

list_source_files async

list_source_files(
    eval_set_id: str,
    prefix: str | None = None,
    start_after: str | None = None,
    exclude_dirs: Sequence[str] = (),
) -> tuple[list[SourceFileInfo], str | None]

List an eval set's .eval logs with size and S3 ETag, or with prefix (artifacts/<sample_uuid>/) one page of that sample's artifacts, minus files under directories named in exclude_dirs.

Returns the files and the start_after of the next page, None when complete.

sign_source_files async

sign_source_files(paths: list[str]) -> dict[str, str]

Presign <eval_set_id>/<file> log and artifact paths; returns path -> URL.

list_files async

list_files(
    run_id: str,
    *,
    kind: Literal["eval-set", "scan"] = "eval-set",
    prefix: str = "",
    recursive: bool = True,
) -> AsyncIterator[RunFile]

Yield every file in an eval set's (or, with kind="scan", a scan's) folder.

Covers everything stored there: logs, logs.json, .config.yaml, artifacts/, a running eval's .buffer/, a scan's parquet results. prefix narrows it to paths starting with it, e.g. artifacts/<uuid>/. With recursive=False, only the files directly at prefix (say, the logs at the top of the folder, without walking every artifact); :meth:list_level also gives its subfolders. Files come in key order, a server page at a time.

list_level async

list_level(
    run_id: str,
    *,
    kind: Literal["eval-set", "scan"] = "eval-set",
    prefix: str = "",
) -> tuple[list[RunFile], list[str]]

The files and subfolders directly under prefix in a run's folder.

The files are those :meth:list_files yields with recursive=False. Each subfolder is a folder-relative path ending in /, ready to pass back as prefix: prefix="artifacts/" gives artifacts/<uuid>/ for each sample with artifacts, without listing their files.

get_file_download_urls async

get_file_download_urls(
    run_id: str,
    paths: list[str],
    *,
    kind: Literal["eval-set", "scan"] = "eval-set",
) -> AsyncIterator[tuple[str, str]]

Yield (path, url) presigned download URLs for files in a run's folder.

paths are as :meth:list_files returns them. A URL works for about 15 minutes from issue and accepts HTTP Range requests, so a large file can be read in parts without downloading all of it.

download_to_file async

download_to_file(path: str, destination: Path) -> None

GET an API path and write the body to destination.

download_transcript_index async

download_transcript_index(
    destination: Path,
    *,
    eval_set_ids: list[str] | None = None,
) -> None

Write the readable transcript index (Arrow IPC stream) to destination.

destination appears only once the whole stream arrived: Arrow reads a stream cut at a message boundary as a complete, shorter index.

download_scan_export async

download_scan_export(
    scanner_result_uuid: str, destination: Path
) -> str

Download a scan-export CSV.

Returns:

Type Description
str

Server filename from Content-Disposition.

get_sample_metadata async

get_sample_metadata(sample_uuid: str) -> SampleMetadata

Look up where a sample's eval log lives.

list_sample_artifacts async

list_sample_artifacts(
    eval_set_id: str, sample_uuid: str
) -> BrowseResponse

List files in a sample's artifact tree.

get_sample_artifact_file_url async

get_sample_artifact_file_url(
    eval_set_id: str, sample_uuid: str, artifact_path: str
) -> PresignedUrlResponse

Presign one sample artifact file.

fetch_logs async

fetch_logs(
    job_id: str,
    *,
    since: datetime | None = None,
    limit: int | None = 100,
    sort: SortOrder = DESC,
    from_start: bool = False,
) -> list[LogEntry]

Fetch runner logs for a job.

get_job_monitoring_data async

get_job_monitoring_data(
    job_id: str, *, since: datetime | None = None
) -> JobMonitoringData

Pod/status snapshot for a job.

get_eval_set_status async

get_eval_set_status(job_id: str) -> EvalSetStatus

One-shot eval-set progress snapshot.

stream_eval_set_status async

stream_eval_set_status(
    job_id: str,
) -> AsyncIterator[EvalSetStatus]

SSE stream of eval-set status until the connection ends.

stop_eval_set async

stop_eval_set(
    eval_set_id: str,
    *,
    sample_uuid: str | None = None,
    error: bool = False,
) -> None

Stop an eval-set, or one sample when sample_uuid is set.

delete_job async

delete_job(job_id: str) -> Literal['eval set', 'scan']

Delete an eval set or scan, retrying 503s within the client timeout.

Honor Retry-After, defaulting to five seconds. Only 404 falls back to scan.

get_job_status async

get_job_status(job_id: str) -> JobStatusResponse | None

Job status, or None if the job is not found.

HawkClientError

Bases: Exception

Base error for :class:~hawk.client.HawkClient.

Raised for missing config or token, and for connection failures.

HawkAPIError

HawkAPIError(
    status: int,
    message: str,
    *,
    problem: dict[str, Any] | None = None,
)

Bases: HawkClientError

The Hawk API returned a non-success HTTP status.

Attributes:

Name Type Description
status int

HTTP status code.

message str

Error text from the API body or reason phrase.

problem dict[str, Any] | None

The parsed Problem JSON body, when the API sent one.