See which setup writes code that lasts
RepoOps groups each merged pull request by the agent, model, skill and prompt pattern behind it, and scores each group on how much of its code was still standing 30 days later. Every figure prints how many changes are behind it, so a gap across 12 changes does not read like a gap across 1,200.
How to
The RepoOps desktop app tracking the repository, with the Claude Code sessions that wrote its changes on this machine.
Merge commits that carry a Session-Id trailer. Run npx repoops install in the checkout once to add the hook that writes it.
- Open Attribution, then the Scorecards tab.
- Pick a grouping.
Agent, Model, Skill or Prompt pattern. A change lands in every group of a kind it belongs to: its subagents (or main session only), the model each session ran, the skills it invoked (or no skill), and the first prompt's first six words or slash command.
- Read the six figures on each row.
Standing at 30 days is the share of changes at least 30 days old with no revert and no fix naming them inside 30 days. Revert rate is the share a later commit reverted. Cost per change that stuck divides the session cost of priced changes by the ones that stood. Time to fix is the median hours from merge to the first revert or fix. Incidents per 1,000 lines merged counts the incident cases that name a change's merge commit or the deploy of it, over the lines those changes added; with no incident store on this machine it reads not measured. Change failure rate is the share of changes at least 30 days old that a revert or a fix followed within 30 days, or that an incident joined.
- Compare AI-written changes with human ones.
The line above the table splits change failure rate in two. A change is AI-written when its merge commit names a session in a Session-Id trailer, and human when it names none. The line prints the share of changes with no session link beside the two rates, since a session that left no trailer counts as human. The repoops.ai homepage prints the same split for the RepoOps repository itself, with the merge window, the day it was read and both samples.
- Check n and the interval before comparing two rows.
Each figure shows its count and a 95 percent interval. With fewer than 5 changes behind it, the figure reads not enough data and shows no number.
- Turn on brain publishing to see it on the hosted dashboard.
The desktop writes the scorecards for the last 90 days at most once a day before its brain publish. In both apps they are the Scorecards tab on Attribution, grouped by the Score by control: Agent, Model, Skill or Prompt pattern. On repoops.ai the tab reads what the desktops published. Cost per outcome, on Cost, prints the all-changes cost per change that stuck from the same summary.
Use cases
Two models side by side
- Situation
- Most work runs on one model and some on a cheaper one, and you want to know whether the cheaper one's changes last.
- You do
- Open Attribution, then the Scorecards tab, and pick Model under Score by.
- You see
- One row per model with standing, revert rate, cost per change that stuck and time to fix. If the intervals overlap, the data does not separate the two yet.
Which way of asking needed a fix
- Situation
- You suspect one prompt habit leads to changes that need a fix within the month.
- You do
- Pick Prompt pattern.
- You see
- Each pattern's revert rate and time to fix with its count. The footer counts the changes whose prompt was not on this machine, so you read the share against what was measured.
Expected outcome
Next step: The merge commits carry no Session-Id trailer. Run npx repoops install in the checkout; future merges join.
Next step: The session transcripts are not on this machine. Read the scorecards on the machine that ran the sessions.
Next step: Turn brain publishing on in the desktop app and wait for the next publish.
What else to consider
What counts as a fix
A fix counts only when its subject is a conventional fix naming the change as #N, or its body has a Fixes #N line. A fix described only in prose is missed, so standing can read high. Skills are read from at most 300 transcripts. An incident counts once for each change its merge commit or deploy names, so a deploy that carried two changes puts its incident on both.
Acceptance by source
What people kept, on the desktop app's Scorecards tab of Attribution, answers a different question: of what RepoOps proposed, how much did you keep. Proposals, drafts, Ask answers, actions, lessons and other sources get an outcome row once their fate is observable, such as a proposal accepted into a brain file or a draft left to go stale, and each source shows one acceptance rate across every tracked repository. A source with nothing decided shows no rate. Rows opens a source's outcomes, each linked to what it scored, and Run tracker checks proposals and session drafts now instead of at the nightly pass. The outcome rows live in each repository's brain on the desktop, so on repoops.ai the same tab, where /team/outcomes redirects, says they are read on the desktop app.
Cost, egress and storage
No model call; every figure is counted. With brain publishing on, the summary goes to the team with group names, counts, intervals and costs, redacted on the way out and again on the way in, and never a diff, transcript or full prompt. The hosted store keeps the latest snapshot per repository. Turn brain publishing off to stop sending; the desktop panel keeps working.