Hardening High-Risk Evals¶

Some evals run code you actively expect to be hostile: cyber and offensive-security tasks, jailbreak and misuse probes, or any run where the point is to see how far a capable model will go.

For those runs you want the strictest isolation Hawk can give you, and you should not have to know what any of it is called.

Say it's high-risk, get everything¶

One field on your eval-set config:

isolation: strict

Every isolation control Hawk has turns on at once. In plain terms:

You get	Meaning
A stronger sandbox boundary	Agent syscalls hit a userspace kernel, not the host's
An unprivileged container	No new privileges, no Linux capabilities, seccomp pinned
No network at all	No internet, no DNS, no cloud metadata, no other evals, no cluster
Nothing borrowed from the host	No host devices, namespaces, or container runtime access
Its own everything, per attempt	Fresh pod, disk, DNS resolver and network policy each sample, destroyed after

Two rules matter as much as the controls themselves:

Deny by default. Leaving something out means locked down, not "unset, so allowed".
Fail loudly. If a control can't be applied, the run stops and tells you. It will never quietly downgrade to a weaker sandbox while your config still says strict.

Just want gVisor?¶

If you only want the syscall boundary on an otherwise ordinary task, name the runtime class and change nothing else:

isolation:
  runtime_class: gvisor

This works at any level and is the researcher-facing way to run a task under gVisor without owning its sandbox definition. It overrides a runtime class the task pinned for itself, so you get the boundary whether or not the task author thought about it. strict implies gvisor already, so you only need this when you are not using strict, or when your cluster offers a stronger runtime under another name.

Mixed eval sets¶

An eval set usually isn't uniform: most tasks could be locked down, a few need the internet. Set the level per task item, and the rest of the set is unaffected:

isolation: standard # the floor for everything below

tasks:
  - package: git+https://github.com/METR/inspect-metr-task-bridge@v0.5.24
    name: mtb
    items:
      - name: bridge
        args: { image_tag: audio_classification-2.1.11 }
        sample_ids: [macaques_numpy]
        isolation: strict # this one is locked down
      - name: bridge
        args: { image_tag: iclr_authors-2.0.1 }
        sample_ids: [notable-top-25]
        # needs the internet, so it stays at the eval set's level

It has to be per item, not per task name, because a bridged eval set gives every item the same task name and distinguishes them by args and sample_ids.

A task item can only make isolation stronger, never weaker: naming a level below the eval set's is rejected. Otherwise a single permissive item would be able to pull the run's guarantee down, and — more to the point — you would be tempted to lower the whole eval set to accommodate it, losing the isolation on everything else.

The opt-outs¶

Real engagements need holes in the wall. Each one is asked for by name, so it stays visible in the config and on the finished run.

Network access¶

Name your targets. Don't reach for network_mode: bridge, which grants *, meaning every domain on the internet.

isolation:
  level: strict
  allow_domains: ["target.example.com"]
  allow_cidr: ["10.20.0.0/24"]

Opening egress widens what the agent can reach outside the cluster. It does not make in-cluster things (other pods, nodes, the Kubernetes API) reachable, because those are separate identities that world doesn't cover. So the only question is how much of the internet the agent gets. Answer it narrowly.

Provider-executed tools¶

Some model providers offer tools that run on their infrastructure rather than in your sandbox: web search, code execution, remote MCP connectors. Because the work happens on the provider's side, your sandbox's network rules are not what governs it.

Treat enabling one as a form of granting network access, and leave them off for a high-risk eval unless the eval is specifically about them.

A GPU¶

gVisor cannot pass GPUs through, so a GPU task runs under the ordinary container runtime and cannot have the syscall boundary. If a GPU task also runs untrusted code, treat the weaker runtime as a known risk and compensate elsewhere (no internet, tighter monitoring, human approvals), or run it on dedicated infrastructure.

An image that can go further¶

Two controls are not in strict by default, because both need an image built for them and most aren't: running as a non-root user, and a read-only root filesystem. If your image is prepared, turn them on:

isolation:
  level: strict
  non_root: true # image declares a non-root USER
  read_only_root: true # image has scratch volumes at /tmp and the working dir

non_root: true on an image whose configured user is root will be refused by kubelet at container creation, and read_only_root: true without writable scratch means the task cannot write where it expects to. Both fail every sample rather than degrading, so turn them on only once the image supports them.

Human oversight¶

Isolation bounds where the agent can go. It does not gate what it does. For high-risk runs, pair it with human-in-the-loop approvals (acp_server / approval), so a person reviews tool calls live and can end a run that exceeds scope. hawk stop ends a run gracefully; hawk delete tears it down immediately.

Before you run¶

Launch one small run first before scaling up, and check it with hawk status and hawk logs.
Check the sandbox can't reach anything you didn't intend.
Decide whether the run needs approvals, and who is watching it.

Reference: the controls underneath¶

You don't need any of this to use isolation, which is set on the eval-set config and applies to every sandbox in the run. This section is for task authors who want to know what the level does to their sandbox, and for working out why a pod was rejected.

What `strict` sets¶

Shown as the Kubernetes sandbox values it produces:

services:
  default:
    image: my-registry/task-image:v1
    runtimeClassName: gvisor # syscall boundary; omit ONLY for GPU tasks
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: RuntimeDefault
      # The two below are `non_root` / `read_only_root`, off by default because
      # they need an image built for them.
      runAsNonRoot: true
      runAsUser: 1000
      readOnlyRootFilesystem: true
    # readOnlyRootFilesystem needs writable scratch, or the task will fail
    volumes:
      - name: tmp
        emptyDir: {}
      - name: work
        emptyDir: {}
    volumeMounts:
      - name: tmp
        mountPath: /tmp
      - name: work
        mountPath: /home/agent
# no allowDomains / allowEntities / allowCIDR -> no internet

Both image-dependent controls need a prepared image

read_only_root requires writable scratch: without emptyDir volumes at /tmp and at the agent's working directory, the task cannot write where it expects to and even pip install fails. Adjust the working-directory path to match your image.

non_root requires an image whose configured user is not root; kubelet refuses to start one that is.

Both fail every sample rather than degrading, which is why they are opt-in.

non_root and Inspect's injected tools

Inspect installs text_editor, bash_session and similar by exec'ing into the sandbox as root, so that their files are root-owned and hidden from the agent. Under non_root that exec cannot succeed: switching users needs CAP_SETGID, which strict drops.

Inspect falls back to installing them as the sandbox's own user, so the tools still work. The tools directory is then readable by the agent rather than hidden from it. Worth knowing if your eval depends on the agent being unable to inspect its own tooling; it does not otherwise affect the run.

Without non_root the sandbox already runs as the user Inspect asks for, so no switch is needed and nothing is given up.

Control by control¶

Control	Default	Harden it
Syscall isolation	runc (no gVisor)	`runtimeClassName: gvisor`
User	root (uid 0)	`runAsNonRoot` + `runAsUser` (opt-in: `non_root`)
Linux capabilities	default set (no `CAP_SYS_ADMIN`)	`capabilities.drop: [ALL]`
Privilege escalation	permitted	`allowPrivilegeEscalation: false`
Syscall filtering	runtime default profile (active)	pin `seccompProfile: RuntimeDefault`
Root filesystem	writable	`readOnlyRootFilesystem: true` + `emptyDir` scratch (opt-in: `read_only_root`)
Internet	denied	keep it denied, or scope it
Sibling services	reachable	`network_mode: none` per service

Anything the agent must not reach (a scorer, a reference solution, a hidden test set) goes in its own service with network_mode: none, which is deny-all even while the agent service has internet. The trusted harness still reaches it over the Kubernetes API, so scoring keeps working.

The short version: at standard, a sandbox's network is locked down by default and its runtime is not. Hardening then lives in the task's own sandbox definition, so you can harden one task by hand without changing anything globally.

What the baseline already gives you, before any hardening

In a default sandbox pod:

No internet, no DNS resolution of external names, no reach to cloud metadata (IMDS), the VPC, or the Kubernetes API.
A seccomp filter is active (the container runtime's default profile).
CAP_SYS_ADMIN is not held, the capability most container escapes need. mount and user-namespace creation both fail.
No container-runtime socket, no host PID or mount namespace, no host devices, no writable cgroup interface.
SELinux enforcing with per-container MCS categories, on an immutable, minimal host OS.

So the baseline is not weak. The hardening above closes what's left, and is worth doing when the code is actively hostile.

What is not on by default: gVisor is off unless the task names runtimeClassName: gvisor, and the container runs as root with privilege escalation permitted and a writable root filesystem.

What gVisor actually changes

The same checks under both runtimes:

	runc	gVisor
Kernel serving your syscalls	the real host kernel	`4.19.0-gvisor`, a userspace kernel
`/proc/kallsyms`	populated	empty
`/sys/kernel/debug`	present	absent

A kernel exploit written for the host kernel now lands on gVisor's reimplementation instead of the host. That is the whole point of the boundary.

The same task image under two settings, seen from inside the sandbox:

	`isolation: {runtime_class: gvisor}`	`isolation: strict`
`uname -r`	`4.19.0-gvisor`	`4.19.0-gvisor`
`CapEff`	`00000000a80405fb`	`0000000000000000`
Egress	blocked	blocked

Both get the boundary. Only strict drops every capability, which is the difference between naming a runtime class and asking for the whole posture.

gVisor can look less isolated if you measure it naively. Inside a gVisor sandbox some interfaces that are locked down under runc appear open: creating a user namespace succeeds, /proc/sysrq-trigger is writable, and Seccomp reads as 0. None of these are regressions. They are gVisor's emulated interfaces. Writing its sysrq-trigger never reaches the host, and a user namespace created inside gVisor is gVisor-internal. Seccomp inside is unnecessary because your syscalls never reach the host kernel directly, and runsc is itself seccomp-confined on the host side.

This matters if you are scoring a breakout eval: judge whether the host was reached, not whether a dangerous-looking interface is visible. A checklist that counts visible primitives will rank gVisor below runc, which is backwards.

Your deployment has to offer a hardened runtime

strict uses whichever runtime class the deployment configures as its hardened one (hawk:sandboxRuntimeClassName, commonly gvisor), rather than assuming a particular one — so a deployment standardising on something else still gets its boundary.

A deployment that configures none cannot provide the boundary at all, so strict is refused when you submit rather than running without it. Naming one yourself with isolation.runtime_class also works, and takes precedence.

Operators: you can make gVisor the default instead

hawk:enableGvisor only makes the runtime available. Pods still have to name it, so on its own it changes nothing about sandbox isolation. Setting hawk:sandboxRuntimeClassName: gvisor makes gVisor the default for every sandbox that doesn't pin its own runtime, so the guarantee no longer depends on each task author remembering.

Worth considering for a cluster dedicated to cyber or otherwise untrusted evals. GPU services are exempt automatically, since gVisor has no GPU passthrough, so they stay on the node default rather than becoming unschedulable.

Full network detail is in Sandbox Network Isolation.