Prove · Error-analysis workbench
Error-analysis workbench
Read your agent runs one at a time and mark each pass or fail. Open a run, inspect its span waterfall, tag a failure mode with a one-line critique, and watch the failure modes accrete into a frequency table that tells you what to fix first.
Where to find it
- Localhost:
/agent-workbench.html?repo=<id> - API:
GET /api/agent/quality?repo=<id>(runs),/api/agent/labels(labels),/api/agent/failure-clusters(taxonomy),/api/agent/traces/<sessionId>(waterfall) - Keyboard: ⌘ K then
workbench - Sidebar: Loop Engineering → Error-analysis workbench
What it does for you
Runs, priciest first.The center column lists your agent sessions sorted by cost, with the source tool, tool count, and any PR, CI, or revert chips. Reading the expensive runs first is how you find what is worth fixing.
Span waterfall per run.Click a run to see its trace as a nested waterfall (session, turn, tool), each span sized by cost or token count, so you can spot the tool call or turn that ran up the bill.
Label pass or fail in one keystroke.Press
p for pass, f for fail, write a one-line critique, and press Enter to save. On a fail you pick a failure mode, or leave it and the critique is auto-categorized.Failure modes rank themselves.Your labels accrete bottom-up into a "fix this first" taxonomy, ranked by frequency, with an honest coverage line (how many sessions are labeled) that warns when the sample is thin.
Judge alignment.Once runs have both a human label and a quality score, the panel shows how well the automated judge agrees with you: TPR, TNR, accuracy, and a confusion matrix.
Save a run as a regression test.The Save as regression test button snapshots a run's trace into a golden dataset (
routing-smoke) that the eval gate keeps checking.Configure
The taxonomy runs deterministically with no key. The optional Merge similar (BYOK) toggle folds near-duplicate clusters using your own API key; it is off by default and fail-closed, so with no key the deterministic taxonomy shows unchanged.