Forensics · Reviewer calibration

Calibrate review outcomes

Count a later defect as a reviewer miss only when it is attributable to the revision and scope the reviewer actually saw.

For: engineers investigating AI-assisted work, reviewers deciding what the evidence establishes, and team owners checking the same record across devices

What it does, and why it helps

Calibration partitions later signals into potential miss, confirmed miss, unrelated failure, disputed, or insufficient evidence. Reopening a case is not an adjudication, and a hosted signal without line attribution stays potential rather than scoring a reviewer down.

The pain. Naive metrics blame reviewers for environment failures and code added after approval.

The point of view. A review is comparable only inside its recorded revision, scope, and criteria.

What gets easier. Discussing misses with evidence instead of a misleading percentage.

When it helps. A defect appears after a reviewed AI-assisted change.

Its limits. Hosted records without attributed lines cannot become confirmed misses.

Where to find it

  • Desktop: AI incidents, open a case and its review evidence.
  • Hosted: /team/agent-incidents, case reviewer calibration rows.
  • API: GET /api/ai-incident-cases/:id

When to use it

Adjudicate a later defect

Situation. An AI-assisted change needs an answer that another reviewer can reproduce.

What you do. Compare the approved digest and scope with the recurrence's attributed lines and record the adjudication.

What you see. The row names the class, reason code, review id, signal id, and evidence fields.

What it establishes. Only a comparable, attributed, relevant defect contributes a confirmed miss.

Before you start

Supported versions
RepoOps v0.3.1, verified against the 2026-09-17 completion worktree.
Where it runs
The local view needs a tracked repository. The team view needs a bound device and an authenticated team membership.
Permissions
Local evidence follows the repository's own access. Hosted reads are tenant and repository scoped before rows, counts, filters, or exports are built.
Connections
Local inspection works offline. Team delivery needs the device binding and repoops.ai; a failed delivery keeps its cursor and reports the failure.
Plan
Local deterministic analysis is available without a paid runner. Hosted compute, private runners, and model analysis are disabled by default and never silently substituted.

Configure it

  1. Record approvals with exact revision and file scope.

    This establishes the scope before any conclusion is computed.

  2. Adjudicate later signals as relevant, not relevant, or disputed.

    The receipt records the decision and the evidence ids it used.

SettingWhereA sensible choiceWhy it matters
ⓘ
To stop or undo
Withdraw a mistaken attribution or append a new adjudication; never edit the earlier row.

What you should see

Insufficient evidence

Configuration. Use a tracked repository and leave every unknown or disputed input in its real state.

Expect. The later signal remains in the report but is excluded from a confirmed-miss numerator.

Verify. The reason names the missing revision, scope, attribution, or adjudication.

Data and cost

What is captured
Approval digest, scope, criteria, actor, later-signal ids, attributed paths and revisions, and adjudications.
Who can see it
Team-scoped report rows contain pointers and bounded reasons, not transcript bodies.
How long it is kept
Approval and case records follow team/case deletion; the report is derived.
What leaves the machine
No external model or reviewer service is called.
What it costs
Deterministic and unmetered.

When the result differs

SymptomLikely causeNext action
A clear defect reads potential miss.Hosted evidence has no attributed lines or relevant adjudication.Add the missing attribution from the owning device and have a person adjudicate it.
Disable
Stop the relevant capture or prevention toggle in Settings; already recorded evidence remains a historical record.
Roll back
Restore the earlier package version or withdraw the disputed edge. Append the correction; do not rewrite the earlier decision.
Revoke access
Revoke the device or transcript grant. Later reads become restricted while identifiers and non-secret receipts remain auditable.
Delete
Delete the owning case or team through its deletion flow. Prevention groups also expire coherently after 365 days of inactivity.
Work an AI incidentcase, evidence, attribution, fix, preventionBuild and verify a remedybounded local execution and explicit approvalReview data boundariescapture, sharing, retention, and deletion

Maintenance evidence

Feature id
reviewer-calibration (spine leaf reviewer-calibration)
Owner
Forensics runtime and public-story owners (LDG-0695 through LDG-0699, LDG-0722)
Supported product version
RepoOps v0.3.1
Last verified
2026-09-17
Example fixtures
Deterministic unit and integration fixtures cited by the source modules; external acceptance gates remain named separately.
Source references
lib/reviewer-calibration.mjs, website/lib/ai-incident-cases/reviewer-calibration.ts, website/lib/ai-incident-cases/open-case.ts
Documentation review
Claims checked against the implementation and dated evidence. Unknown, disputed, restricted, not-run, and unavailable states are retained.
Video review
No licensed rendered story is published for this guide yet. The guide does not render a placeholder player.

Last updated