Distributed Tracing¶

Hawk ships OpenTelemetry tracing for the API, the eval runner, and the middleman, exported directly to AWS X-Ray (SigV4-signed OTLP, no collector or sidecar). When enabled you can follow a request from the CLI through the API into the runner, middleman, and out to the LLM provider, and pull up every span belonging to an eval set with a single query.

Enabling¶

Set otelTracingEnabled: "true" in your Pulumi stack config. This:

enables tracing in the API (HAWK_API_OTEL_TRACING_ENABLED), runner, and middleman (HAWK_OTEL_TRACING_ENABLED),
enables X-Ray Transaction Search so span attributes are queryable,
grants the runner and middleman roles xray:PutTraceSegments.

otelSpanSamplingPercentage controls Transaction Search indexing (default 100).

The trace model¶

A Hawk eval is one trace, end to end:

The CLI mints an X-Amzn-Trace-Id (one per CLI invocation, no AWS credentials needed) and sends it on API requests, so the API's request span joins a CLI-originated trace.
The API passes its request span's context to the runner pod via the HAWK_OTEL_TRACE_PARENT env var; the runner's eval_set.run / scan.run span is parented under it. Runner retries re-read the same value and join the same trace.
The runner injects X-Amzn-Trace-Id into every model request, so middleman and provider-call spans nest inside.

The X-Ray trace map therefore draws the whole chain as one connected component: Client → hawk-api → hawk-runner → hawk-middleman → provider. Shared annotations on every span (see below) are the query mechanism — one Transaction Search query returns everything for an eval set. Note that a trace spanning a very long eval is subject to X-Ray's maximum trace duration (7 days).

Span hierarchy¶

└── POST /eval_sets/            hawk-api      (hawk.job.id, hawk.user.*)
    └── eval_set.run            hawk-runner   (hawk.job.id, hawk.user.*)
        └── sample              hawk-runner   one per sample (inspect.sample.uuid, …)
            ├── sample.execution hawk-runner  solver/agent phase (waiting_time_s)
            ├── sample.scoring  hawk-runner   scoring phase
            ├── sample.heartbeat hawk-runner  every 60s while running: live waiting/working time + tokens
            ├── model.usage     hawk-runner   per successful call: tokens, retries, cumulative usage
            ├── tool            hawk-runner   per tool call: name, working_time, failed
            └── model.generate  hawk-runner   per model call (peer.service=<provider>)
                └── POST /…     hawk-middleman (hawk.job.id, inspect.sample.uuid, …)
                    └── upstream.request hawk-middleman (upstream.provider, peer.service, status, latency)

Timelines group one collapsible sample row per sample, with its phases and model calls nested inside, rather than interleaving phase spans across samples.

Span attributes¶

Attribute	Where	Meaning
`hawk.job.id`	all services	eval_set_id / scan_run_id
`hawk.job.type`	runner, middleman	`eval-set` or `scan`
`inspect.eval_set.id`	API, runner, middleman	eval set id (eval sets only)
`inspect.eval.id`	runner, middleman	inspect eval id
`inspect.sample.uuid`	runner, middleman	inspect `EvalSample.uuid`, matches warehouse sample records
`inspect.sample.epoch`, `inspect.sample.id`	runner	epoch and dataset sample id
`hawk.user.id`, `hawk.user.email`	all services	who launched the job
`inspect.usage.*_tokens`, `inspect.usage.retries`	runner (`model.usage`)	per-call token usage and HTTP retries; the span itself covers the successful call's duration
`inspect.sample.*_tokens`	runner (`model.usage`, `sample`)	cumulative token usage for the sample (running total on each `model.usage` span; final total on the `sample` span)
`inspect.sample.waiting_time_s`	runner (`sample.heartbeat`, `model.usage`, `sample.execution`, `sample`)	cumulative time the sample spent waiting on rate limits / retries / concurrency — heartbeats include waits still in progress
`inspect.sample.working_time_s`	runner (`sample.heartbeat`)	live working time (elapsed minus waiting)
`inspect.tool.name`, `inspect.tool.working_time_s`, `inspect.tool.failed`	runner (`tool`)	per tool call — name, active time, outcome. Arguments/results are never recorded. Disable with `HAWK_OTEL_TOOL_SPANS=0`.
`service.version`	all services	deployed image URI
`upstream.provider`, `peer.service`	middleman	LLM provider of an upstream call (drawn as a downstream node in the trace map)

Model names are intentionally not recorded on spans; middleman scrubs model names from Gemini-style request paths before they reach span names.

Finding traces¶

Everything for one eval set — in the CloudWatch console, Application Signals → Transaction Search, query:

attributes.hawk.job.id = "<eval-set-id>"

This returns the submission trace and the eval trace together. Filter by attributes.hawk.user.email to see a user's activity, or attributes.inspect.sample.uuid to find the model calls and middleman hops of a single sample.

Live samples — while a sample is running, the runner emits a sample.heartbeat span every 60 seconds with the sample's cumulative rate-limit waiting time (including a wait currently in progress), working time, and token usage so far. To check on a long-running sample, query the most recent heartbeat:

name = "sample.heartbeat" and attributes.inspect.sample.uuid = "<uuid>"

Trace map / timeline — open any matching trace in the X-Ray console to see the service map (api → runner → middleman) and the per-span timeline: how long a sample spent executing vs scoring, each model.generate call, and the middleman/provider latency underneath it.

Datadog — if Datadog ingestion is configured, the same spans appear under APM as a single trace per eval.

Notes¶

Trace headers are not forwarded to LLM providers; provider latency is measured by the middleman's upstream.request client span instead.
Health-check requests (and the CLI's unauthenticated /auth/config bootstrap) are never traced.
If the exporter can't determine an AWS region or credentials, tracing degrades gracefully: spans are created but not exported, and the service keeps running.