Sample Editing¶
You can submit batched corrections to samples that have already been imported into the warehouse: override a score, mark a sample as invalid, or attach extra events to a transcript. Every edit is addressed by sample_uuid, and edits are submitted together in a JSON or JSONL file with hawk edit-samples.
Collect the sample UUIDs¶
Run hawk list samples against the eval set and take the value in the UUID column:
Write the edits file¶
Each edit is an object with a sample_uuid and a details payload. Use a JSON array or JSONL, whichever you prefer:
[
{
"sample_uuid": "<sample-1-uuid>",
"details": {
"type": "score_edit",
"scorer": "includes",
"value": "C",
"reason": "Grader missed a correct answer phrased as a range."
}
},
{
"sample_uuid": "<sample-2-uuid>",
"details": {
"type": "invalidate_sample",
"reason": "Sandbox lost network at turn 12; the transcript is not gradeable."
}
}
]
Choose the edit type¶
Set details.type to whichever of the five operations you need:
type |
Use it to |
|---|---|
score_edit |
Corrects a score a grader already produced, creates a new score if scorer doesn't exist. |
invalidate_sample |
Marks a sample as invalid. |
uninvalidate_sample |
Brings an invalidated sample back into analysis. |
append_info_event |
Attaches a note to the transcript, leaving scores alone. |
append_score_event |
Records a score as a transcript event. It does not change the reported scores or metrics. |
score_edit¶
Corrects a score a grader already produced. scorer (the score's name) and reason are required; value, answer, explanation and metadata each default to "UNCHANGED", so name only the fields you are actually changing. value is mandatory if scorer doesn't exist yet and takes whatever shape the scorer produces: a string such as "C", "I", "P" or "N", a number, a boolean, a list, or a dict of those:
{
"type": "score_edit",
"scorer": "model_graded_qa",
"explanation": "Rewritten after manual review; value stands.",
"reason": "Explanation referenced the wrong rubric item."
}
A metadata dict replaces the score's metadata, it does not merge into it.
If you pass metadata, repeat any keys the scorer set that you want to keep.
invalidate_sample¶
Marks a sample as invalid and records your name and reason. It does not remove the sample from the eval's metrics. Filter on it yourself (the Invalid column in the viewer). If the eval set is run again, invalidated samples are re-run from scratch. reason is required:
uninvalidate_sample¶
Reverses an invalidation. Takes no other fields:
append_info_event¶
Attaches a note to the transcript, leaving scores alone. source labels where the note came from, and data is any JSON you want to carry:
{
"type": "append_info_event",
"source": "manual-review",
"data": { "reviewer": "alice", "verdict": "borderline" }
}
append_score_event¶
Records a score as an event in the sample transcript. It does not add the score to the sample's reported scores, and it does not change the eval metrics. This is also true when intermediate is false. To add a reported score, use score_edit with a new scorer name.
scorer and score are required. Set intermediate to true if the score should not be treated as final:
{
"type": "append_score_event",
"scorer": "human_review",
"score": {
"value": "P",
"explanation": "Reached the goal but broke a rule."
},
"intermediate": false
}
A batch must hold between 1 and 1000 edits, and the same operation may not appear twice against the same slot in one batch
Two score_edits for the same sample and scorer, or two invalidate_samples, or two uninvalidate_samples on the same sample, are rejected as duplicates. Different edit types on the same sample (including an invalidate followed by an uninvalidate) are accepted and applied in the order listed. Append operations are exempt: you can append several events of the same source or scorer to one sample in a single batch.
Submit the file¶
Hawk checks the whole batch before it stores any edit. The checks cover the file format, your permissions, and duplicate edits. If a check fails, Hawk rejects the request and applies nothing.
A failure after these checks can leave some edits queued. Hawk stores a batch that spans several eval logs as one job for each log. If the request returns an error, some of these jobs may already be stored. Hawk can still apply them. Before you submit the batch again, check the affected samples with hawk transcript <sample-uuid>.
Accepted edits are then applied to the eval logs, so a change may take a minute to appear. Acceptance is not success: an edit can still fail while being applied, and you will not be notified, so check that it landed in the sample's transcript:
The Scores table lists one row per scorer, which is enough to confirm a score_edit that changed value, answer or explanation. Every other trace an edit leaves is in the sample JSON, which --raw gives you: