Python client¶
Scripts can call the Hawk HTTP API without installing the CLI (no Click, no Inspect).
The page below is generated from HawkClient docstrings at docs-build time.
Install¶
After a release that includes the client:
From a checkout, without publishing:
Task authors can read declared input identities and locations through
hawk.runtime.transcript_sources().
This runner helper is separate from the HTTP client and needs no client extra.
Auth¶
HawkClient never opens a browser.
Pass token=, or set HAWK_ACCESS_TOKEN, or reuse a token stored by hawk login on this machine.
Pass api_url=, or set HAWK_API_URL in the environment, a project .env, or
~/.config/hawk-cli/env. Supplied tokens (including token callables) use this same
URL resolution and do not read the keyring.
With HAWK_ACCESS_TOKEN set, the stored login is never read: the environment is
the whole token store (the hawk CLI also reads HAWK_REFRESH_TOKEN beside it;
HawkClient does not).
For saved credentials, the selected deployment supplies its own login, and each
request checks that the API recipient was approved during hawk login.
Supplied tokens go wherever the API URL points
Tokens you supply (token=, a token callable, or HAWK_ACCESS_TOKEN) are not
checked against a saved login: the client sends them to whatever API URL it
resolves, including one from a project .env. The CLI refuses this combination;
the SDK accepts it. If you export a token, set HAWK_API_URL beside it or
pass api_url=, and don't run SDK scripts from untrusted checkouts.
Unrelated service settings do not invalidate an API login. After upgrading from
client-ID-scoped storage, run hawk login once per deployment.
Dotenv values are literal; ${VAR} interpolation is disabled.
Authenticated API requests do not follow redirects; configure the final API URL.
The client does not read or refresh a stored hawk login refresh token.
Pass refresh_token= when submitting jobs that use user credentials so the runner can refresh on the cluster.
With credential_mode="restricted", submission and resume requests omit the refresh token even if supplied.
With user or omitted mode, they forward it; an omitted mode lets the server choose.
The access token still authenticates every submission.
Dict-typed responses (get_usage, get_usage_history, and listing TypedDicts) are the server JSON. Fields may be added without a client major bump.
import asyncio
from hawk.client import HawkClient
async def main() -> None:
async with HawkClient() as client:
jobs = await client.get_jobs(mine=True, limit=5)
print(jobs)
asyncio.run(main())
On a machine with no prior hawk login, you can pass the URL and token explicitly:
Submit an eval-set¶
Pass YAML as a dict. The server validates it. You do not need Inspect installed.
import asyncio
import pathlib
import ruamel.yaml
from hawk.client import HawkClient
async def main() -> None:
yaml = ruamel.yaml.YAML(typ="safe")
config = yaml.load(pathlib.Path("eval-set.yaml").read_text())
async with HawkClient() as client:
created = await client.create_eval_set(config, priority="high")
print(created["eval_set_id"], *created["warnings"], sep="\n")
asyncio.run(main())
Optional local validation uses EvalSetConfig from hawk.core.types.evals (still no Inspect for typical configs).
create_scan accepts a mapping the same way. create_scan_detailed and
resume_scan_detailed return the same warnings list as part of the response. The
warnings name package sources that can change between runs; the launch has already
started when they arrive.
create_eval_set, create_scan, and resume_scan accept priority="low",
"medium", or "high" for admission and inference. Evals default to medium;
scans default to low. Each explicit resume uses its current argument or that
default. Priority is a client argument, separate from the eval/scan YAML config.
Read a run's files¶
list_files yields every file in an eval set's folder (or a scan's, with
kind="scan"), and get_file_download_urls presigns them. With
recursive=False it yields only the files directly under prefix (by default
the top of the folder, where the logs are) without walking artifacts/.
list_level returns that level's files and its subfolders, each ending in /,
from one listing: with prefix="artifacts/", one subfolder per sample that has
artifacts. A URL is valid for
about 15 minutes and answers GET, including HTTP Range requests, so a
script can read the start of a large file without downloading all of it. A
reader that runs longer, or one that sends HEAD first, needs new URLs or a
downloaded copy:
import asyncio
import aiohttp
import yarl
from hawk.client import HawkClient
async def main() -> None:
async with HawkClient() as client, aiohttp.ClientSession() as http:
files = [f async for f in client.list_files("my-eval-set", recursive=False)]
evals = [f for f in files if f["path"].endswith(".eval")]
# Agents write these names, so print them escaped.
print(repr(max(evals, key=lambda f: f["size_bytes"])["path"]))
_, folders = await client.list_level("my-eval-set", prefix="artifacts/")
for folder in folders:
print(repr(folder)) # 'artifacts/<sample-uuid>/'
async for _path, url in client.get_file_download_urls(
"my-eval-set", ["logs.json"]
):
# As signed: aiohttp would otherwise re-quote characters in the key.
signed = yarl.URL(url, encoded=True)
async with http.get(signed, headers={"Range": "bytes=0-65535"}) as response:
if not response.ok: # aiohttp's own error would print the signed URL
raise SystemExit(f"logs.json: HTTP {response.status}")
print(len(await response.read()))
asyncio.run(main())
Scans with scanner models¶
With scanner_models (see
Model scanners over everything you can read),
one request launches several scans. create_scan returns the first launched one's
id, and raises HawkClientError naming each scan's error if none launched; others may
still have failed, so use create_scan_detailed to check each outcome.
create_scan_detailed returns the whole response as a ScanCreated: siblings
holds one ScanSibling per scan (scan_run_id, scanner_model, evals,
transcripts, outcome, and for a failed scan error and resumable), next to
excluded_evals, not_imported_eval_sets and forbidden_eval_sets. Both types
are exported from hawk.client.
A refusal raises HawkAPIError, whose problem holds the parsed Problem JSON.
When some evals can't be scanned, problem["code"] is "unscannable_evals" and
problem["unscannable"] lists every one (eval_set_id, file unless it is a
whole eval set, reason, models).
Delete a job¶
await client.delete_job(job_id) waits for asynchronous restricted cleanup by
retrying HTTP 503 responses. It honors Retry-After (seconds or an HTTP date),
using five seconds when the header is absent or invalid. Requests and waits share
the client's timeout budget, which defaults to 300 seconds. Other errors are
reported immediately; only a 404 tries the scan endpoint after the eval endpoint.
The hawk delete CLI command uses the same retry behavior.
API¶
HawkClient ¶
HawkClient(
*,
api_url: str | None = None,
token: str | Callable[[], str] | None = None,
timeout: float = _DEFAULT_TIMEOUT_SECONDS,
)
Async HTTP client for the Hawk API.
Use as async with HawkClient(...) as client:. Requests without entering
the context manager raise HawkClientError.
Pass api_url and token, or omit them to read HAWK_API_URL and
reuse a token from HAWK_ACCESS_TOKEN / hawk login. With
HAWK_PROFILE set and no token, requests use that restricted
profile's token instead, never the full login's.
Does not prompt for login or refresh a stored login token; the CLI owns that.
Submission and resume methods omit refresh_token when
credential_mode="restricted". User or omitted mode forwards it for runner
credential renewal; an omitted mode lets the server choose.
Open a client.
Token and URL are resolved when entering async with, not here.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
api_url
|
str | None
|
Hawk API base URL. Defaults to |
None
|
token
|
str | Callable[[], str] | None
|
Bearer token, or a callable returning one, called for every
request (for tokens that rotate) in a worker thread
( |
None
|
timeout
|
float
|
aiohttp total timeout in seconds for ordinary requests (default 300). Downloads and SSE streams use their own timeouts. |
_DEFAULT_TIMEOUT_SECONDS
|
create_eval_set
async
¶
create_eval_set(
eval_set_config: EvalSetConfig
| Mapping[str, Any]
| None = None,
*,
eval_set_id: str | None = None,
image: str | None = None,
image_tag: str | None = None,
secrets: dict[str, str] | None = None,
log_dir_allow_dirty: bool = False,
refresh_token: str | None = None,
allow_sensitive_cross_lab_scan: bool = False,
force: bool = False,
credential_mode: CredentialMode | None = None,
priority: JobPriority = DEFAULT_EVAL_PRIORITY,
) -> EvalSetCreated
Create an eval-set, or resume one by passing eval_set_id.
Accepts an EvalSetConfig or a config mapping. Pass None with an
existing id to resume the saved config. Priority defaults to medium for
every submission, including resumes.
allow_sensitive_cross_lab_scan permits private transcript inputs to
be sent to another provider when the deployment operator allows it. It
applies only to this submission and must be re-provided on resume.
Returns:
| Type | Description |
|---|---|
EvalSetCreated
|
The eval-set id, plus any advisory warnings the server raised |
EvalSetCreated
|
about the launch (empty against a server that predates them). |
create_scan
async
¶
create_scan(
scan_config: ScanConfig | Mapping[str, Any],
*,
image: str | None = None,
image_tag: str | None = None,
secrets: dict[str, str] | None = None,
refresh_token: str | None = None,
allow_sensitive_cross_lab_scan: bool = False,
force: bool = False,
credential_mode: CredentialMode | None = None,
priority: JobPriority = DEFAULT_SCAN_PRIORITY,
) -> str
Create a Scout scan from a ScanConfig or mapping.
Priority defaults to low. Returns its scan run id; with scanner_models,
the first launched sibling scan's, raising HawkClientError if none
launched (create_scan_detailed lists them all).
create_scan_detailed
async
¶
create_scan_detailed(
scan_config: ScanConfig | Mapping[str, Any],
*,
image: str | None = None,
image_tag: str | None = None,
secrets: dict[str, str] | None = None,
refresh_token: str | None = None,
allow_sensitive_cross_lab_scan: bool = False,
force: bool = False,
credential_mode: CredentialMode | None = None,
priority: JobPriority = DEFAULT_SCAN_PRIORITY,
) -> ScanCreated
As create_scan, returning the whole response: with scanner_models,
the sibling scans launched and the evals and eval sets left out.
upload_selection
async
¶
Upload a file of sample uuids (one per line) and return its selection id.
resume_scan
async
¶
resume_scan(
scan_run_id: str,
*,
image: str | None = None,
image_tag: str | None = None,
secrets: dict[str, str] | None = None,
refresh_token: str | None = None,
allow_sensitive_cross_lab_scan: bool = False,
force: bool = False,
credential_mode: CredentialMode | None = None,
priority: JobPriority = DEFAULT_SCAN_PRIORITY,
shards: int | None = None,
) -> str
Resume from the last checkpoint with this submission's priority.
Priority defaults to low, regardless of the original run.
resume_scan_detailed
async
¶
resume_scan_detailed(
scan_run_id: str,
*,
image: str | None = None,
image_tag: str | None = None,
secrets: dict[str, str] | None = None,
refresh_token: str | None = None,
allow_sensitive_cross_lab_scan: bool = False,
force: bool = False,
credential_mode: CredentialMode | None = None,
priority: JobPriority = DEFAULT_SCAN_PRIORITY,
shards: int | None = None,
) -> ScanCreated
As resume_scan, returning the whole response with its warnings.
get_jobs
async
¶
get_jobs(
*,
mine: bool = True,
limit: int = 10,
status: JobStatus | None = None,
waiting: bool = False,
job_ids: Sequence[str] | None = None,
) -> list[JobListItem]
List jobs (eval-sets and scans).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mine
|
bool
|
If true, only jobs created by the token's |
True
|
limit
|
int
|
Max rows. |
10
|
status
|
JobStatus | None
|
Optional status filter. |
None
|
waiting
|
bool
|
If true, only jobs with pending human interactions. |
False
|
job_ids
|
Sequence[str] | None
|
Only these jobs (at most 100 per request). |
None
|
get_eval_sets
async
¶
List eval-sets.
get_evals
async
¶
List evals in an eval-set.
get_samples
async
¶
get_samples(
eval_set_id: str,
*,
search: str | None = None,
page: int = 1,
limit: int = 50,
) -> list[SampleListItem]
List one page of samples in an eval-set.
get_all_samples_for_eval_set
async
¶
get_all_samples_for_eval_set(
eval_set_id: str, *, limit: int | None = None
) -> list[SampleListItem]
Walk sample pages until exhausted, or until limit rows.
search_transcripts
async
¶
search_transcripts(
query: str,
*,
eval_set_id: str | None = None,
sort: Literal["rank", "recency"] = "rank",
limit: int = 50,
cursor: str | None = None,
diversify: bool = True,
time_limited: bool = False,
) -> SearchResultPage
One page of full-text transcript search (GET /meta/search).
Pass the returned next_cursor back as cursor, with the same query,
scope, sort and diversify, until it is None. Cursors expire after one
hour. diversify=False lifts the viewer's per-transcript hit cap.
time_limited=True answers within seconds instead of failing when the
index is slow: such a page can be an unranked, newest-first scan, often
empty but with a cursor, and its cursor needs time_limited=True too.
The scan skips attachment text and ignores diversify.
stream_search_csv
async
¶
Stream every full-text match as CSV (GET /meta/search.csv).
One request; the server walks the pages and flushes each as rows. If
it stops early the last line is #resume-cursor=<cursor>; resume with
search_transcripts(..., sort="recency", diversify=False, cursor=...).
A query too broad to rank exports only its 10,000 newest matches and
then ends with #truncated=too-broad,limit=10000, which nothing resumes.
If such an export fails on a later page it ends with #truncated=error
instead: no cursor resumes it and the rows are incomplete. Callers must
handle both #truncated= lines (the request opts in to them) and treat
any reason but too-broad as a failed export.
get_me
async
¶
The caller's identity as the server resolves it.
Returns the GET /auth/me JSON: sub, email (the claim named
by the deployment's email field, or None), exp, is_admin,
model_groups (each with read, write and execute flags)
and code_groups, as this deployment enforces them.
create_delegated_session
async
¶
Start a delegated session for a restricted profile and get its first token.
Needs a full OIDC login; delegated tokens are refused.
fetch_delegated_session_token
async
¶
Mint a new token for one of the caller's delegated sessions.
Raises HawkAPIError with 404 (unknown session), 410 (session ended)
or 403 (nothing left to grant).
list_delegated_sessions
async
¶
The caller's active delegated sessions.
delete_delegated_session
async
¶
End one of the caller's delegated sessions; its tokens stop being renewed.
get_usage
async
¶
Current token usage, grouped by provider and model.
group_by selects the split: "user" (the default, filling each
model's users), "team" (teams), "job" (jobs) or
"task" (tasks). The parameter is only sent when it differs from
the default, so servers predating it see the same request as before.
Returns the server JSON list. Fields may be added without a client major bump.
get_usage_history
async
¶
get_usage_history(
start: int,
end: int,
bin_seconds: int,
group_by: str = "user",
) -> list[dict[str, Any]]
Binned token usage between Unix timestamps start and end.
group_by selects the split: "user" (the default, filling each
bin's users), "team" (teams), "job" (jobs) or
"task" (tasks). The parameter is only sent when it differs from
the default, so servers predating it see the same request as before.
Returns the server JSON list. Fields may be added without a client major bump.
get_log_files
async
¶
List .eval log files for an eval-set.
get_log_headers
async
¶
Fetch partial eval-log headers for the given file names.
get_download_url
async
¶
Presign one log file.
Returns:
| Type | Description |
|---|---|
tuple[str, str]
|
|
get_download_urls
async
¶
Yield presigned S3 URLs for log files as each batch completes.
Batches of BATCH_DOWNLOAD_URLS_LIMIT paths are posted concurrently.
list_source_files
async
¶
list_source_files(
eval_set_id: str,
prefix: str | None = None,
start_after: str | None = None,
exclude_dirs: Sequence[str] = (),
) -> tuple[list[SourceFileInfo], str | None]
List an eval set's .eval logs with size and S3 ETag, or with prefix
(artifacts/<sample_uuid>/) one page of that sample's artifacts, minus files
under directories named in exclude_dirs.
Returns the files and the start_after of the next page, None when complete.
sign_source_files
async
¶
Presign <eval_set_id>/<file> log and artifact paths; returns path -> URL.
list_files
async
¶
list_files(
run_id: str,
*,
kind: Literal["eval-set", "scan"] = "eval-set",
prefix: str = "",
recursive: bool = True,
) -> AsyncIterator[RunFile]
Yield every file in an eval set's (or, with kind="scan", a scan's) folder.
Covers everything stored there: logs, logs.json, .config.yaml,
artifacts/, a running eval's .buffer/, a scan's parquet results.
prefix narrows it to paths starting with it, e.g. artifacts/<uuid>/.
With recursive=False, only the files directly at prefix (say, the
logs at the top of the folder, without walking every artifact);
:meth:list_level also gives its subfolders. Files come in key order, a
server page at a time.
list_level
async
¶
list_level(
run_id: str,
*,
kind: Literal["eval-set", "scan"] = "eval-set",
prefix: str = "",
) -> tuple[list[RunFile], list[str]]
The files and subfolders directly under prefix in a run's folder.
The files are those :meth:list_files yields with recursive=False.
Each subfolder is a folder-relative path ending in /, ready to pass
back as prefix: prefix="artifacts/" gives artifacts/<uuid>/
for each sample with artifacts, without listing their files.
get_file_download_urls
async
¶
get_file_download_urls(
run_id: str,
paths: list[str],
*,
kind: Literal["eval-set", "scan"] = "eval-set",
) -> AsyncIterator[tuple[str, str]]
Yield (path, url) presigned download URLs for files in a run's folder.
paths are as :meth:list_files returns them. A URL works for about
15 minutes from issue and accepts HTTP Range requests, so a large
file can be read in parts without downloading all of it.
download_to_file
async
¶
GET an API path and write the body to destination.
download_transcript_index
async
¶
Write the readable transcript index (Arrow IPC stream) to destination.
destination appears only once the whole stream arrived: Arrow reads a
stream cut at a message boundary as a complete, shorter index.
download_scan_export
async
¶
Download a scan-export CSV.
Returns:
| Type | Description |
|---|---|
str
|
Server filename from |
get_sample_metadata
async
¶
Look up where a sample's eval log lives.
list_sample_artifacts
async
¶
List files in a sample's artifact tree.
get_sample_artifact_file_url
async
¶
get_sample_artifact_file_url(
eval_set_id: str, sample_uuid: str, artifact_path: str
) -> PresignedUrlResponse
Presign one sample artifact file.
fetch_logs
async
¶
fetch_logs(
job_id: str,
*,
since: datetime | None = None,
limit: int | None = 100,
sort: SortOrder = DESC,
from_start: bool = False,
) -> list[LogEntry]
Fetch runner logs for a job.
get_job_monitoring_data
async
¶
Pod/status snapshot for a job.
get_eval_set_status
async
¶
One-shot eval-set progress snapshot.
stream_eval_set_status
async
¶
SSE stream of eval-set status until the connection ends.
stop_eval_set
async
¶
Stop an eval-set, or one sample when sample_uuid is set.
delete_job
async
¶
Delete an eval set or scan, retrying 503s within the client timeout.
Honor Retry-After, defaulting to five seconds. Only 404 falls back to scan.
get_job_status
async
¶
Job status, or None if the job is not found.
HawkClientError ¶
Bases: Exception
Base error for :class:~hawk.client.HawkClient.
Raised for missing config or token, and for connection failures.
HawkAPIError ¶
Bases: HawkClientError
The Hawk API returned a non-success HTTP status.
Attributes:
| Name | Type | Description |
|---|---|---|
status |
int
|
HTTP status code. |
message |
str
|
Error text from the API body or reason phrase. |
problem |
dict[str, Any] | None
|
The parsed Problem JSON body, when the API sent one. |