Eval Set Config Reference¶
Fields and nested config models for eval-set YAML, generated from the
EvalSetConfig Pydantic schema. Task/model/agent package entries
(PackageConfig[...], BuiltinConfig[...]) are omitted here — see
Running Evaluations for those. For checkpoint
behaviour and resume, see Checkpointing & Resume.
Top-level fields (EvalSetConfig)¶
| Field | Type | Default | Description |
|---|---|---|---|
tags |
list[str] | None | None |
Tags to associate with this run. |
metadata |
object | None | None |
Metadata to associate with this run. Can be specified multiple times. |
runner |
RunnerConfig | (model defaults) | Configuration for the runner. |
name |
str | None | None |
Name of the eval set config. If not specified, it will default to 'eval-set'. |
eval_set_id |
str | None | None |
The eval set id. If not specified, it will be generated from the name with a random string appended. Max 43 chars to fit K8s namespace limits. Must contain only lowercase alphanumeric characters and hyphens, and must start and end with an alphanumeric character. |
packages |
list[str] | None | None |
List of other Python packages to install in the sandbox, in PEP 508 format. |
tasks |
list[PackageConfig_TaskConfig_] | required | List of tasks to evaluate in this eval set. |
models |
list[PackageConfig_ModelConfig_ | BuiltinConfig_ModelConfig_] | None | None |
List of models to use for evaluation. If not specified, the default model for each task will be used. |
model_cost_config |
dict[str, ModelCostConfig] | None | None |
Costs to use (input/output/cache read/cache write) for each specified model, in dollars and per 1M tokens. |
model_roles |
dict[str, SingleModelPackageConfig | SingleModelBuiltinConfig] | None | None |
Named roles for use in get_model(). |
solvers |
list[PackageConfig_SolverConfig_ | BuiltinConfig_SolverConfig_] | None | None |
List of solvers to use for evaluation. Overrides the default solver for each task if specified. |
agents |
list[PackageConfig_AgentConfig_ | BuiltinConfig_AgentConfig_] | None | None |
List of agents to use for evaluation. Overrides the default agent for each task if specified. |
approval |
str | ApprovalConfig | None | None |
Config file or object for tool call approval. |
acp_server |
int | None | None |
TCP loopback port for the in-eval Inspect ACP server; enables human-in-the-loop tool approvals reachable via the Hawk relay (hawk acp <run_id>). When unset, ACP is disabled and behaviour is unchanged. |
approval_timeout_minutes |
float | None | 10080 |
Minutes a parked ACP approval waits before auto-rejecting. Requires acp_server. Defaults to one week; null waits indefinitely. |
score |
bool | True |
Whether to score model output for each sample. If False, use the 'inspect score' command to score output later. |
limit |
int | list[Any] | None | None |
Evaluate the first N samples per task, or a range of samples [start, end]. |
sample_shuffle |
bool | int | None | None |
Shuffle order of samples (pass a seed to make the order deterministic). |
epochs |
int | EpochsConfig | None | None |
Number of times to repeat the dataset (defaults to 1). Can also specify reducers for per-epoch sample scores. |
message_limit |
int | None | None |
Limit on total messages used for each sample. |
token_limit |
int | None | None |
Limit on total tokens used for each sample. |
time_limit |
int | None | None |
Limit on clock time (in seconds) for each sample. |
working_limit |
int | None | None |
Limit on total working time (e.g. model generation, tool calls, etc.) for each sample, in seconds. |
cost_limit |
float | None | None |
Limit on total cost (in dollars) for each sample. |
retry_attempts |
int | None | None |
Maximum number of times inspect_ai.eval_set will retry a failed task (defaults to inspect-ai's default of 10). Set to 0 to disable retries. |
log_realtime |
bool | True |
Log events in realtime (enables live viewing of samples in inspect view). |
log_model_api |
bool | True |
Log raw model api requests and responses. Note that error requests/responses are always logged. |
log_images |
bool | None | None |
Log base64-encoded images in the eval log. Defaults to inspect-ai's default (True) when unset. |
adaptive_connections |
bool | int | False |
Enable inspect-ai's adaptive connections controller, which automatically scales model API concurrency based on rate-limit feedback. Set to an integer N to also raise the controller's concurrency ceiling to N (inspect-ai shorthand for AdaptiveConcurrency(max=N)); true uses inspect-ai's default bounds. Note that an explicit per-model max_connections takes precedence over the adaptive controller (adaptive is silently disabled for that model), so remove max_connections for adaptive to apply. May become the default in a future release. See https://inspect.aisi.org.uk/models-concurrency.html#adaptive-connections. |
monitor |
bool | False |
Enable Datadog monitoring for this eval set. When enabled, a log-based monitor will alert via Slack DM if errors appear during execution. |
isolation |
IsolationConfig | None |
How locked-down this eval set's sandboxes are. standard (the default) is whatever the deployment's baseline provides. strict is for code you expect to be hostile: a gVisor syscall boundary, an unprivileged read-only container, and no network egress at all. Set it on its own (isolation: strict) or as a mapping to name exceptions. |
checkpoint |
CheckpointConfig | None |
Sample-level checkpointing config. Off by default. Set checkpoint.enabled: true to periodically snapshot in-progress samples to durable storage so a crashed run can be resumed with hawk eval-set resume. |
human_eval |
HumanEvalOverrides | None | None |
Overrides for the human-eval rewrite path. Ignored for non-human eval-sets and for --no-rewrite human evals. |
secrets |
list[EnvSecretConfig | AwsSecretsManagerSecretConfig] | [] |
List of required secrets/environment variables that must be provided by the user |
Extra keys not listed here are allowed and passed through to inspect_ai.eval_set().
RunnerConfig¶
Configuration for the runner that executes the evaluation.
| Field | Type | Default | Description |
|---|---|---|---|
image |
str | None | None |
Full container image URI for the runner (e.g., 'ghcr.io/org/runner:v1'). Must include an explicit tag or digest. The ':latest' tag is not allowed. If not specified, the default runner image from the platform config is used. |
image_tag |
str | None | None |
Tag within the runner Docker image repository to use for the runner. If not specified, the API's configured default will be used. |
memory |
str | None | None |
Memory limit for the runner pod in Kubernetes quantity format (e.g., '8Gi', '16Gi'). If not specified, the API's configured default will be used. Setting this also reserves that much memory on the node. A deployment may schedule runners that leave this unset against a smaller request, but a runner that names its memory is scheduled against the value it names, so raise it only as far as the job needs: reserving more memory fits fewer runners per node. |
cpu |
str | None | None |
CPU limit for the runner pod in Kubernetes quantity format (e.g., '2', '4'). If not specified, the API's configured default will be used. |
cleanup |
bool | None | None |
Whether to clean up the runner and sandbox environments after the eval completes. Set to false to keep them alive for debugging. Use hawk delete to clean up manually. |
secrets |
list[EnvSecretConfig | AwsSecretsManagerSecretConfig] | [] |
List of required secrets/environment variables that must be provided by the user |
environment |
dict[str, str] | {} |
Environment variables to set for the job. Should not be used to set sensitive values, which should be set using the secrets field instead. |
oom_diagnostics_enabled |
bool | False |
When true, the runner starts an always-on memray allocation tracker and uploads a diagnostics bundle (memray + py-spy stacks + cgroup state) to s3:// |
ModelCostConfig¶
| Field | Type | Default | Description |
|---|---|---|---|
input |
float | required | Price per million input tokens. |
output |
float | required | Price per million output tokens. |
input_cache_write |
float | required | Price per million input tokens written to cache. |
input_cache_read |
float | required | Price per million input tokens read from cache. |
ApprovalConfig¶
| Field | Type | Default | Description |
|---|---|---|---|
approvers |
list[ApproverConfig] | required | List of approvers to use. |
EpochsConfig¶
| Field | Type | Default | Description |
|---|---|---|---|
epochs |
int | required | Number of times to run each sample. |
reducer |
str | list[str] | None | None |
One or more functions that take a list of scores for all epochs of a sample and return a single score for the sample. |
IsolationConfig¶
Sandbox isolation for an eval set. Accepts the level on its own (isolation: strict) or as a mapping when exceptions are needed.
| Field | Type | Default | Description |
|---|---|---|---|
level |
'standard' | 'strict' | 'standard' |
Isolation level to run every sandbox in this eval set at. All levels may get more restrictive with new releases. |
allow_domains |
list[str] | None | None |
Domains the sandbox may reach at strict, which otherwise permits no egress at all. Prefer naming targets over granting the whole internet. |
allow_cidr |
list[str] | None | None |
CIDR ranges (e.g. 10.20.0.0/24) the sandbox may reach at strict. |
runtime_class |
str | None | None |
Kubernetes RuntimeClass to run every sandbox service under, e.g. gvisor for a syscall-level boundary. Defaults to gvisor at strict, and to whatever the deployment provides at standard. Setting it strengthens the sandbox, so it is accepted at any level, and it overrides a runtime class the task pinned for itself. |
allow_gpu |
bool | False |
Let services that request a GPU run at strict without the gVisor syscall boundary, which has no GPU passthrough. Off by default so a task cannot opt itself out of the boundary just by asking for a GPU. |
non_root |
bool | False |
Also require the sandbox to run as a non-root user at strict. Off by default because it needs an image that declares a non-root USER; kubelet refuses to start one that doesn't, so turning this on without preparing the image fails every sample. |
read_only_root |
bool | False |
Also require a read-only root filesystem at strict. Off by default because it needs writable scratch volumes mounted at /tmp and the agent's working directory; without them a task cannot write where it expects to and fails. |
CheckpointConfig¶
Configuration for sample-level checkpointing. When enabled, in-progress samples are periodically snapshotted (host Inspect state + the declared in-sandbox paths) to durable storage, so a crashed runner can resume them via hawk eval-set resume instead of restarting them from scratch. Checkpoints only fire for an agent or solver that integrates (ticks) Inspect's checkpointer. metr_agents/react is one such agent, but any agent or solver can add the same support; one that does not tick the checkpointer will not produce checkpoints even when this is enabled. Capturing in-sandbox paths requires the sandbox to permit root exec (Inspect injects a restic binary as root). Sandboxes that block root exec will fail samples; keep checkpointing off to run without it.
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | False |
Whether checkpointing is on for this eval-set. Off by default; can be enabled on any eval-set. Only takes effect with an agent or solver that ticks Inspect's checkpointer (otherwise nothing is snapshotted), and requires a sandbox that permits root exec or samples crash at start. |
trigger |
CheckpointTriggerConfig | None |
When to fire checkpoints. Defaults to every 10 minutes. |
sandbox_paths |
dict[str, list[str]] | None | None |
Eval-wide override of the per-sandbox-name absolute paths to capture inside the sandbox. Leave unset (the default) so tasks declare their own checkpoint sandbox_paths per sample. When set, this acts as an override: per Inspect's merge precedence (eval > sample > task) it REPLACES any task- or sample-declared sandbox_paths wholesale -- so only set it for runs where no task self-declares, or it clobbers their capture. When unset everywhere, checkpoints capture host state only. |
max_consecutive_failures |
int | None | None |
Abort the sample after this many consecutive failed checkpoint writes. Unset (the default) tolerates failures indefinitely, so a sample can finish with no usable checkpoint; set a small value (e.g. 3) to fail fast instead. |
checkpoints_location |
str | None | None |
Override the durable location for checkpoint data (any fsspec-resolvable path, e.g. 's3://...'). Defaults to a ' |
HumanEvalOverrides¶
Overrides consumed by the human-eval rewrite path. Read by POST /human_evals/ when rewrite=True (the default). Ignored for regular eval-sets (POST /eval_sets/) and for human evals submitted with --no-rewrite -- in either case the user owns the full agent spec.
| Field | Type | Default | Description |
|---|---|---|---|
agent_args |
object | None | None |
Args to set on the operator-configured default human agent. Shallow merge; user-supplied keys win. Example: {user: root, record_session: false}. |
EnvSecretConfig¶
Configuration for a required secret/environment variable.
| Field | Type | Default | Description |
|---|---|---|---|
type |
str | 'env' |
|
name |
str | required | Name of the environment variable. |
description |
str | None | None |
Optional description of what this secret is used for. |
AwsSecretsManagerSecretConfig¶
Configuration for a required secret from AWS Secrets Manager.
| Field | Type | Default | Description |
|---|---|---|---|
type |
str | 'aws-secrets-manager' |
|
name |
str | required | Name of the environment variable. |
arn |
str | None | None |
Optional full AWS Secrets Manager ARN to source the secret from. |
secret_name |
str | None | None |
Optional Secrets Manager secret name to source the secret from, resolved under the deployment's default prefix exactly like name is. Use it to inject one env var (name) from a differently-named secret (e.g. name=HF_TOKEN, secret_name=team-x/HF_TOKEN). Mutually exclusive with arn. |
description |
str | None | None |
Optional description of what this secret is used for. |
ApproverConfig¶
Configuration for an approval policy that Inspect can look up by name.
| Field | Type | Default | Description |
|---|---|---|---|
name |
str | required | Name of the approver to use. |
tools |
list[str] | required | These tools will need approval from the given approver. |
args |
object | None | None |
Approver arguments, passed as keyword arguments to the approver. May not contain the keys 'name', 'tools' or 'params', which Inspect reserves for the structure of an approver entry. |
CheckpointTriggerConfig¶
When to fire a checkpoint during a sample. Maps to one of Inspect's checkpoint trigger specs. Checkpoints are only evaluated at agent turn boundaries, so the effective cadence is max(turn duration, the configured interval).
| Field | Type | Default | Description |
|---|---|---|---|
type |
'time' | 'turn' | 'token' | 'manual' | 'time' |
Trigger strategy: 'time' fires after a wall-clock interval, 'turn' every N agent turns, 'token' every N tokens of sample usage, 'manual' only on explicit agent checkpoint() calls. |
every |
int | None | None |
Interval for the trigger: seconds for type='time', number of turns for type='turn', number of tokens for type='token'. Ignored for type='manual'. Defaults to 600 (10 minutes) when type='time' and left unset. |