How we prove the accountability numbers
The three published numbers come from real RepoOps code run over a committed synthetic fixture. Same fixture in, same numbers out, recorded in a checked-in results file. Reproducible math, not numbers we made up. A live version that runs the same metrics against your own repo is coming.
By the numbers
The three headline numbers, reproducibly
These are the same numbers shown on the homepage. The fixture and the code paths are committed, so the numbers are fixed: the same fixture produces the same result every time, recorded in a checked-in file.
9 of 12 seeded defects traced to the exact session that caused them, at the proof-counting confidence bands.
12 of 12 known bugs blocked at the pre-merge gate, out of 20 review findings.
The tracked session cost the prevented recurrences would have re-run: 6 priced, 2 unpriced (no cost signal), across 12 prevention events. Sourced from the committed results file, never estimated.
What the output looks like
One command, one readable table
Inside the repo, npm run bench:accountability produces this table and writes the committed results file at docs/benchmarks/accountability-results.json. The numbers above come straight from that file.
$ npm run bench:accountability [bench] RepoOps accountability benchmark (fixture: accountability-v1) [bench] ---------------------------------------------------------- [bench] Attribution accuracy : 9/12 = 75% (bands: exact 3, strong 6, weak 1, none 2) [bench] Recurrence prevented : 12/12 = 100% (of 20 review findings) [bench] Measured $ saved : $24.55 (priced 6, unpriced 2, of 12 prevention events) [bench] ---------------------------------------------------------- [bench] wrote docs/benchmarks/accountability-results.json
How it works
What the benchmark computes
What the benchmark does
It loads the documented fixture, a synthetic repo of sessions, commits, defects, and fixes, and runs the real RepoOps accountability code paths over it: the same git-blame attribution, the same pre-merge recurrence gate, and the same savings pricing the product uses. It runs inside a throwaway temporary brain, so it never touches real data.
What each number means
- Attribution accuracy: how many defects were traced back to the right session, split into exact, strong, weak, and none confidence bands.
- Recurrence prevented: how many known bugs the pre-merge gate blocked from shipping again, out of the review findings it saw.
- Measured dollars saved: the real spend those prevented recurrences would have cost, priced from actual token usage.
Reproducible by design
The fixture, the code paths, and the results file are all committed, so the numbers are fixed and auditable: the same fixture always produces the same result. The claim is checked in, not asserted.
Coming: your own numbers
- The fixture proves the math. Next, run the same metrics against your own repo.
- repoops bench, from the installed app, runs the same attribution, prevention, and savings code over your real git history, telemetry, and brain.
- It will sign you in and show your numbers in the terminal and the hosted dashboard. A repo with no RepoOps telemetry gets real git attribution today and lights up prevention and savings once you connect RepoOps.
Why it is helpful
- Audit the claim, not the slide: the fixture, code paths, and results file are all committed, so the number is reproducible math.
- Confirm the numbers did not regress across a RepoOps change: the committed results file records exactly what was produced.
- Give your team or leadership a reproducible basis for the accountability story.
Inside the repo the harness is scripts/bench-accountability.mjs and the committed results file is docs/benchmarks/accountability-results.json. A live per-repo repoops bench is coming.