Tutorial 3 of 7
Release gates for a new prompt or model version
Intermediate 12 min
Compare two prompt versions on eval sets with McNemar tests, batch-validate structured outputs and turn your quality bar into gates that pass or fail.
Should prompt v2 ship? Open the Prompt v2 release regression template. It holds two eval sets, a batch of invoice-extraction outputs and a trace of a weather agent.
1. Eval sets
Select QA and math set. The Regression eval check takes CSV or JSONL with id, input, expected, output_a, output_b and an optional category; repeated ids are repeated runs. Name the versions in Name of version A / B and pick a Scoring metric (exact, normalized text, contains, numeric with tolerance, regular expression).
The result: prompt-v2 58.33 % vs prompt-v1 83.33 % on 12 items: 4 regressions, 2 improvements. The chart shows accuracy per category (geo and math both drop). The finding explains the statistics honestly: prompt-v2 scores 25 points below prompt-v1; with McNemar p = 0.6875 the drop is not statistically significant yet. Twelve items are simply too few — add more before you trust either version. The table Regressions lists the items v2 got wrong; near misses such as “Mount Kilimanjaro” are flagged as possible formatting differences.
2. Structured outputs in a batch
Select Invoice extraction. Paste the JSON Schema once and many model outputs below it, separated by a line that contains only --- (or one JSON object per line). The help text counts the outputs as you type.
Run check makes one Structured Output Validator call per output and shows a progress bar. The result is 3 of 5 outputs match the schema (60 %): one total came back as a string, one date as 03.10.2026. Download batch results (CSV) gives you the table per output for a bug report.
3. Define the gates
Open Gates (key 3). The template uses four gates:
| Gate | Threshold | Result |
|---|---|---|
| Accuracy change of the new version (points) | ≥ −2 | −25: fail |
| Statistically significant eval drops | ≤ 0 | 0: pass |
| Structured outputs matching the schema | ≥ 95 % | 60 %: fail |
| Loops in traces | ≤ 0 | 1: fail |
Each gate is evaluated over the latest successful run of every matching check, and the worst value decides — the column Deciding check names it. Change a threshold, switch a gate off with its checkbox, Add a gate (19 metrics are available, from High-severity findings to Requests reaching the expected tool in the top 3) or Reset to the default gates. Gate edits are undoable (Ctrl+Z).
4. Iterate
Fix the prompt, paste the new outputs and press Run all (Ctrl+Shift+Enter). The toolbar chip turns green (Gates pass) when every gate passes. If a gate has no data, the verdict says so instead of passing silently.
Use Edit → Duplicate check (Ctrl+D) to keep the old eval next to the new one, then compare both in the Compare tab: metric deltas plus new and resolved findings.
5. Share the decision
Export → Release gates (CSV) and Report (Markdown) put the verdict into your pull request or release ticket. The project file (.agentlab.json) keeps the full history for the next release.
Ready to try it?