Skip to content

Re-importing eval logs

When the eval-log importer changes what it writes to the warehouse, existing rows only change once their .eval log is imported again. Each Hawk stack has a Step Functions state machine, <env>-hawk-eval-log-reimport, that force re-imports a list of logs. It runs in AWS under its own service role, so a long run doesn't depend on anyone's laptop or SSO session. A person with admin access to the account (e.g. ProductionAdmin) starts it by hand. Nothing starts it automatically.

How it works

  • You upload a JSON list of items to the stack's re-import bucket (eval_log_reimport_bucket_name stack output) under lists/. The bucket is separate from the inspect logs bucket, so uploads can't trigger the importer's S3 event rules. Objects expire after 90 days. The lists name internal eval sets, so never commit them to the repo.
  • A Distributed Map submits one importer Batch job per item, with --force true, to the on-demand importer queue. It waits for each job (submitJob.sync) and never has more than max_concurrency in flight. Items whose size is at most the stack's evalLogImporterSmallMaxBytes (default 10 MiB; 0 disables downsizing) get the same small Fargate tier the importer Lambda uses.
  • Results (succeeded and failed items, with the error) go to results/<map-run-id>/ in the same bucket.
  • Re-imports are idempotent, so stopping, redriving or re-running a list is safe.

The list file is a JSON array:

[{"key": "evals/<eval-set-id>/<file>.eval", "size": 123456}, {"key": "evals/..."}]

The execution input looks like this. Only list_key is required:

{"list_key": "lists/<name>.json", "max_concurrency": 8, "tolerated_failure_percentage": 0.5}

Choosing concurrency

The cap is the only throttle. Unlike a hand-run driver, Step Functions does not back off when the warehouse is busy. The warehouse, not Batch, is the bottleneck: the default of 8 has been comfortable, and the state machine rejects values above 32. Raise it with care. Watch RDS DBLoad (Performance Insights / Datadog) for the first part of a run. If the warehouse is struggling, lower the cap of the running Map Run:

aws stepfunctions list-map-runs --execution-arn <execution-arn>   # -> mapRunArn
aws stepfunctions update-map-run --map-run-arn <map-run-arn> --max-concurrency 4

Jobs already in flight keep running, so the load drops only as they finish. The alternative is to stop the execution and start a new execution, with a lower cap, of the items that haven't succeeded. Don't redrive to lower the cap: a redrive reuses the original input, including max_concurrency.

Start a run

Before a large run, tell whoever is on call: failed items land in the importer's Batch DLQ and trip its alarm.

With the helper (it reads stack outputs and records each log's size). It needs a Pulumi login to the stack's state backend (see the repo's AGENTS.md):

export AWS_PROFILE=<admin-profile>   # e.g. ProductionAdmin for prd
pulumi login "s3://<pulumi-state-bucket>?region=<region>&awssdk=v2"
uv run python scripts/ops/start-eval-reimport.py \
    --stack prd --keys-file keys.txt --name importer-names-2025-06 \
    --max-concurrency 8 [--dry-run]

keys.txt has one evals/... key or s3://<logs-bucket>/evals/... URI per line. The helper checks --max-concurrency and --tolerated-failure-percentage and stops if it can't read a log's size for any reason other than the log not existing, all before uploading anything. Missing logs stay in the list and fail in the run. The helper prints the execution ARN and a console URL. --name becomes both the list's object name and the execution name. The helper refuses to overwrite an existing list, so each run needs a new name. Execution names must be unique per state machine for 90 days.

Or by hand:

aws s3api put-object --bucket <reimport-bucket> --key lists/<name>.json \
    --body items.json --if-none-match "*"   # fails if the name is taken
aws stepfunctions start-execution \
    --state-machine-arn <eval_log_reimport_state_machine_arn> \
    --name <name> \
    --input '{"list_key": "lists/<name>.json", "max_concurrency": 8}'

Watch progress

  • Console: Step Functions → <env>-hawk-eval-log-reimport → the execution → the Reimport Map Run. It shows counts of pending, running, succeeded and failed items.
  • CLI:
aws stepfunctions list-map-runs --execution-arn <execution-arn>
aws stepfunctions describe-map-run --map-run-arn <map-run-arn>   # itemCounts
  • Batch jobs are named <env>-hawk-eval-log-reimport on the <env>-hawk-eval-log-importer-on-demand queue. Their logs go to the importer's usual log group.

Stop, redrive

aws stepfunctions stop-execution --execution-arn <execution-arn>
aws stepfunctions redrive-execution --execution-arn <execution-arn>

Stopping also terminates the in-flight Batch jobs. Redrive re-runs only the items that failed or never ran, and it works for 14 days after the execution ends. A run aborts once more than tolerated_failure_percentage of its items have failed. After you fix the cause, redrive it.

Only a failed, timed-out or aborted execution can be redriven. A run that finished with failures within its tolerance counts as SUCCEEDED and is not redrivable. To retry its failures, start a new execution from the keys in its FAILED_0.json (see below).

Pull out the failures

Results are keyed by the Map Run id, not the execution name. A mapRunArn ends in Reimport:<uuid>; the id is the <uuid> after the last colon. A failed job's Error is States.TaskFailed, and its Cause is the Batch job description.

map_run_arn=$(aws stepfunctions list-map-runs --execution-arn <execution-arn> \
    --query 'mapRuns[0].mapRunArn' --output text)
map_run_id=${map_run_arn##*:}
aws s3 ls s3://<reimport-bucket>/results/${map_run_id}/
aws s3 cp s3://<reimport-bucket>/results/${map_run_id}/FAILED_0.json - \
    | jq -r '.[] | (.Input | fromjson | .key) + "\t" + .Error + "\t" + (.Cause // "" | .[:200])'

To retry the failures as a new run, turn them into a keys file for the helper:

aws s3 cp s3://<reimport-bucket>/results/${map_run_id}/FAILED_0.json - \
    | jq -r '.[].Input | fromjson | .key' > retry-keys.txt

Large runs can write several FAILED_<n>.json files. List the prefix and read them all.

Failed imports also reach the importer's Batch DLQ through its usual failed-job rule, so a run with failures will trip the DLQ alarm, the same as failed imports submitted any other way.

Interaction with the importer's EventBridge rules

  • Spot fallback only matches jobs on the Spot queue, so it ignores these on-demand jobs. A runner job is never resubmitted behind the state machine's back.
  • Failed import → DLQ matches every failed job of the importer job definition on a non-Spot queue, so these failures land in the Batch DLQ (see above). Redrive the execution instead of replaying those DLQ messages, or the import runs twice. That is harmless, just wasteful.