Babysitting Evals (ACP Control Channel)¶

Every running eval can expose a live control channel: the in-eval Inspect ACP (Agent Client Protocol) server, reachable from your machine through the Hawk relay. Over this channel you (or an agent acting for you) can:

Watch a sample's conversation live — messages, tool calls, token usage.
Approve or reject tool calls parked on a human-in-the-loop approval policy.
Steer an interactive sample by sending it messages.
Cancel a runaway tool call or a whole sample.

Humans get a terminal UI (hawk acp launches Inspect's TUI). Agents speak the same protocol programmatically — it's newline-delimited JSON-RPC over a local TCP socket. Both are covered below.

Enabling the control channel¶

Set acp_server in your eval-set config. To make the eval wait for a babysitter on selected tools, add an approval policy with a human approver — those approvals park on the ACP channel until someone answers (or the timeout expires):

# See examples/acp-approval.eval-set.yaml for a complete config.
acp_server: 4444              # loopback port inside the runner pod
approval_timeout_minutes: 60  # parked approvals auto-reject after this (default: one week)

approval:
  approvers:
    - name: human             # park these tool calls for interactive approval
      tools:
        - "bash*"
    - name: auto              # everything else proceeds
      tools:
        - "*"

Without an approval policy the channel is still useful: you can observe any running sample and cancel samples/tool calls — the eval just never blocks waiting for you.

approval_timeout_minutes defaults to one week and requires acp_server; set it to null to wait indefinitely. While a sample is parked it consumes no model tokens, but its sandbox pods stay up.

Attaching as a human¶

hawk acp <run-id>        # defaults to the last job you used

This bridges a loopback port to the run's ACP server through the relay (authenticated with your hawk login token) and launches Inspect's inspect acp TUI against it: a sample picker, live transcript, and approve/reject prompts.

Attaching as an agent (or script)¶

Start the bridge without the TUI, on a known port:

hawk acp <run-id> --no-launch --local-port 4444

The bridge prints ... listening on 127.0.0.1:4444 and keeps running until you Ctrl-C it (run it in the background from scripts). Everything after this point is plain JSON-RPC 2.0, one JSON object per line, over that TCP socket. Anything that can open a socket can babysit an eval — no Hawk or Inspect libraries required.

The in-eval ACP server starts listening shortly after the runner pod reaches Running (package installation happens first), so wait for hawk watch to show the run underway before attaching. If you connect too early, either the connection drops before any JSON-RPC reply (just reconnect and retry) or the relay refuses the attach outright and the bridge exits with a clear error. A relay 404 means there is no live runner pod — the run hasn't started yet (restart the bridge and retry, checking hawk watch) or it has already finished (stop retrying). A 403 means you lack write access to the run's model groups (or the CLI is pointed at a different deployment) — retrying won't help.

Trust model

The loopback port is unauthenticated for the lifetime of the bridge, same as kubectl port-forward. The relay hop itself is authenticated and requires write access to every model group on the run; a run with no model groups is attachable by any authenticated user of the deployment.

A working reference client ships in the repo: examples/acp_babysitter.py (stdlib-only). It attaches to the first live sample, streams updates, and answers approvals:

hawk acp <run-id> --no-launch --local-port 4444 &
python examples/acp_babysitter.py 127.0.0.1:4444          # approve everything
python examples/acp_babysitter.py 127.0.0.1:4444 --deny   # reject everything

The protocol, in five steps¶

1. Initialize (must be first):

{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": 1, "clientInfo": {"name": "my-babysitter", "version": "0.1"}, "clientCapabilities": {}}}

2. Find a sample. inspect/list_samples enumerates all active samples. The pending field is the babysitter's main signal — "approval" means the sample is parked waiting for a tool-call decision:

{"jsonrpc": "2.0", "id": 2, "method": "inspect/list_samples"}

{"samples": [{"sessionId": "MP4a…", "task": "my_pkg/my_task", "sampleId": "0",
  "epoch": 1, "interactive": false, "pending": "approval", "totalTokens": 150, "…": "…"}]}

Samples early in startup can appear with sessionId: null (nothing to attach to yet) — poll until one carries a sessionId.

3. Bind to it with standard ACP session/load:

{"jsonrpc": "2.0", "id": 3, "method": "session/load", "params": {"sessionId": "MP4a…", "cwd": "/", "mcpServers": []}}

On bind the server replays the sample's transcript so far, then streams live session/update notifications: user_message_chunk, agent_message_chunk, tool_call, tool_call_update, and token-usage updates.

4. Answer permission requests. When a parked tool call needs a decision, the server sends you a request (note the id — you must reply to it):

{"jsonrpc": "2.0", "id": 0, "method": "session/request_permission", "params": {
  "sessionId": "MP4a…",
  "toolCall": {"title": "bash echo hello", "rawInput": {"command": "echo hello"}, "…": "…"},
  "options": [{"optionId": "approve", "kind": "allow_once"},
               {"optionId": "reject", "kind": "reject_once"},
               {"optionId": "terminate", "kind": "reject_always"}]}}

Reply with the chosen option:

{"jsonrpc": "2.0", "id": 0, "result": {"outcome": {"outcome": "selected", "optionId": "approve"}}}

The option ids come from your approval policy (approve / reject / terminate for the standard human approver). Your client must handle these server→client requests as well as responses to its own calls.

5. Detect the end. When the sample's agent loop finishes you receive a notification: {"method": "inspect/session_ended", "params": {"sessionId": "…"}}. The connection stays open — list samples again to pick up the next one.

Method reference¶

Standard ACP methods (see the ACP spec):

Method	Direction	Purpose
`initialize`	client → server	Handshake; must be first
`session/load`	client → server	Bind to a sample by `sessionId` (replays transcript, then streams)
`session/new`	client → server	Bind via the picker flow (used by the TUI; agents prefer `session/load`)
`session/prompt`	client → server	Send a user message to an interactive sample's agent
`session/cancel`	client → server	Interrupt the current agent turn
`session/update`	server → client (notification)	Message chunks, tool calls, usage updates
`session/request_permission`	server → client (request)	A parked tool call needs your decision — reply with an outcome

Inspect extensions (the inspect/* namespace):

Method	Direction	Purpose
`inspect/list_samples`	client → server	All active samples, with `pending` / `interactive` / usage fields
`inspect/list_sessions`	client → server	Only ACP-attachable sessions
`inspect/attach`	client → server	Direct bind by `task/sample_id/epoch` target (alternative to `session/load`)
`inspect/cancel_sample`	client → server	Cancel a whole sample (`action`: score what's done vs. raise an error)
`inspect/cancel_tool_call`	client → server	Cancel one in-flight tool call by `toolCallId`
`inspect/session_ended`	server → client (notification)	The bound sample's agent loop finished
`inspect/event`	server → client (notification)	Raw transcript-event firehose (opt in at `initialize` via `_meta: {"inspect.raw_events": ["*"]}`)

interactive: false on a listing means the sample is observe-only — session/prompt is rejected, but watching, approvals, and cancellation all still work. Samples run by Inspect's standard react agent are interactive; custom solvers without a bound agent channel are not.

Babysitting alongside the rest of the CLI¶

The control channel is one tool in the loop an agent uses to drive an eval end-to-end:

Need	Use
Submit / resume	`hawk eval-set …`, `hawk eval-set resume …`
Is it healthy?	`hawk watch --no-follow`, `hawk status`
What is it doing right now?	`hawk acp --no-launch` + ACP, `hawk trace`, `hawk logs -f`
Approve / steer / cancel a sample	ACP (`session/request_permission`, `session/prompt`, `inspect/cancel_sample`)
Why is it stuck?	`hawk stacktrace`, `hawk trace`, debugging guide
Stop / clean up	`hawk stop` (graceful, scores partial work), `hawk delete`
Results	`hawk list samples`, `hawk transcript`, `hawk download`

Local runs¶

hawk local eval-set honors acp_server too — no relay involved. The server listens on the configured loopback port directly, so connect straight to it: inspect acp --server 127.0.0.1:4444, or point a script at that address.