Guard · Incidents
Trace a production alert to the change it blames
Follow a production incident one edge at a time: to the deploy, the commit, the line and the captured session. Every edge names the mechanism that found it and its confidence, a matched session stays a candidate until a person reviews it, and the case shows which evidence was captured and which is missing. An alert whose line cannot be blamed stays unattributed, and the chain runs without a model call.
For: the engineer who owns a production alert, and the person one of these chains names
What it does, and why it helps
A production adapter writes an event into the repository's own event store. The correlator reads the newest of those events, pulls a source file and a line out of the payload, runs git blame on that line, and looks for a captured session whose window covers the commit time and whose recorded files include that file. What comes back is a band, not a verdict. Exact needs the commit to name its session outright, through a Session-Id trailer or a recorded graft, and needs RepoOps to hold that session. Strong is one clean match on time and file. Weak is a rename, a squash, several candidates, or a commit that landed inside the thirty-minute grace window rather than inside the session itself. Unattributed is the floor for an alert with no blameable line, which is every PostHog metric and every slow-query shape by construction.
You read the result on two tabs. Incidents lists one card per problem, newest confidence first, with the source, the band, the implicated file and line, the blamed commit and author, the session, and whether a case exists. Above the cards sit three panels: the seven stages of the accountability loop with the first empty one carrying its note, the production errors that matched a lesson already written, and the measured accuracy of each band. Attribution is the primary page: it lists the open production and performance cases and opens one in the case dialog on its Attribution section, one row per edge with the mechanism that found it, its own confidence (exact, strong, weak or none), and a review state that starts at candidate and moves only when a person moves it. A card offers Save as lesson only on a chain banded strong or exact, and the server refuses a lower one on the write path too.
The edges are incident to deploy, deploy to commit, commit to line, commit to session, session to turn, and commit to developer. Each names its mechanism in words, such as declared by the deployment record, the line's last change, by git blame, an edit receipt from the session or a Session-Id trailer on the commit, no edit observed. There is no pull request edge: the pull request a case shows is its fix. Until someone confirms an edge, the Confirmed cause row reads None. Nothing on this case has been verified as the cause. A change that no agent session touched still has its runtime and GitHub evidence; its coverage says there is no transcript to hold or request.
Every case also carries Evidence coverage: seven evidence families, the transcript among them, each captured, partial, not requested, unsupported, expired or restricted, under a summary such as 3 of 7 evidence families captured. On the hosted Attribution page (/team/agent-traces) an owner or admin of a repository under a managed capture policy can press Retrieve missing evidence. It checks stored content first, then queues a request to the installations linked to the case's sessions for the transcript, tool records and source receipts; no second approval from the developer is asked for. A request reads Queued, Collecting, Received, Partly received, Nothing found, Canceled or Expired, and an offline installation keeps it for up to 72 hours. Evidence already synced stays readable in the session's Captured transcript evidence dialog, which says a verified source means all bytes of that captured snapshot arrived and leaves the rest of the session unverified. The Attribution section's own transcript row still describes the older grant path, a transcript reachable only through a granted request; for a managed repository the retrieval above is that request.
The pain. An alert names a file and a line. Working out which change put that line there, and which session wrote that change, is an archaeology job in git, and the person who finally does it is rarely the person the answer names.
The point of view. A named session is a candidate, not a culprit. Publish the mechanism that found each edge and the confidence it earned, give the person it names a button that says it is wrong, and measure whether the bands deserve their names before anything downstream is built on them.
What gets easier. Reading, and arguing. The card carries the band and the evidence behind it, the case carries one row per edge with the mechanism named, and the appeal sits on the card rather than in a settings page.
When it helps. A repository that is a git checkout, with one production source armed or a crash handler posting to the local receiver, and captured sessions from the last seven days.
Its limits. It cannot blame an alert with no source file. Without a Session-Id trailer on the commit the band stops at strong. A weak band cannot mint a lesson. A blamed commit is the last change to that line, which is not the same as the change that introduced the defect. Nothing here proves causation, and the tab says so.
Understand it in 30 seconds
Read the narration
- 0:00 An alert names a line. Someone has to find the change behind it.
- 0:06 Follow the deploy, the commit and the session.
- 0:09 Each edge names its mechanism and confidence.
- 0:13 The case shows what was captured. A partial transcript reads partial.
- 0:18 A matched session stays a candidate until someone reviews it.
- 0:23 An owner can ask the enrolled installation for the missing transcript.
- 0:27 Offline, the request waits.
Synthetic example. Read the guide
Where to find it
Where to find it
- Desktop:
localhost:4000, then Attribution, one of the six primary pages in the sidebar. - Keyboard: ⌘ K, then type “Attribution”.
When to use it
A Sentry error whose culprit line blames cleanly
Situation. Sentry is armed for the repository, the read token is in the named variable, and the Session-Id hook is installed in the checkout. An error arrives naming a file and a line.
What you do. Open Incidents. Read the card, then press Attribution is right or Not my change and give your name. On a strong or exact card, press Save as lesson, type the fix, and keep or edit the guard pattern the form drafted from the culprit line.
What you see. The header carries the source chip, the band chip and the message. The body reads Implicated file, Change (the short sha and the author), Session, Source link and Case. The draft hint names the file and line it came from and the directory it scoped the pattern to.
What it establishes. The verdict counts toward that band's measured accuracy. The lesson is stored with the fix you typed, and the pattern is stored at warn whatever the page sent, so it reports on a future diff and blocks nobody until a person promotes it.
The chain names the wrong person
Situation. A squash merge collapsed two branches, the band came back weak, and the author on the card did not write the line.
What you do. Press Not my change. Give your name, then the reason if you have one, and the person it really was if you know. Leave that last box empty if you do not.
What you see. The card now reads disputed by your name, with the reason beside it and the replacement name if you gave one. The calibration panel counts your judgement against the band the server holds for that incident, never a band the page chose.
What it establishes. Nothing is rewritten and nothing is re-run. The row stands as the log of what that pass concluded, and the dispute stands beside it. An unnamed replacement stays null, because the rule against inventing an identity holds on the appeal path too.
Before you start
- Supported versions
- RepoOps desktop v0.3.1, the release this guide was read against. The repository has to be a git checkout: the blame is a real git call in your working tree, run with a cleaned environment so a hook's own repository cannot answer for it.
- Where it runs
- Local: Incidents lists the correlated rows; Attribution opens a production or performance case in the case dialog on its Attribution section; the full case page carries the developer's actions. The scope bar's repository applies to the whole page, and environment, service and time narrow the case list only. Hosted: /team/agent-traces is the Attribution page, with the team's cases, the same case sections and Evidence coverage, and the session list, whose panel takes the repository and time but no environment or service. Retrieve missing evidence needs an owner or admin, a repository under a managed capture policy, and the hosted plan.
- Permissions
- No role gate on the desktop. The writes on the tab (a verdict, a saved lesson, a demo event, a prevent step) are same-origin posts to the local server. On a bind that is not loopback, every route except the OAuth-gated MCP surface answers 404.
- Connections
- One of three. A source armed per repository in repos.config.json: sentry, vercel, posthog, dbSlowLog or github, each off by default, each naming the environment variable that holds its token. A Vercel Log Drain, which verifies every batch under a signing secret. Or nothing at all: the local receiver takes a credential-free post from any crash handler, log shipper or curl. Captured sessions are what turn a blame into an attribution, and the lookback is seven days.
- Plan
- No plan gate. The pricing capability map carries no incidents or attribution row. The Lakera Guard card that shares the Incidents tab is Team-gated, and it is a different feature.
Configure it
- Arm a source, or post to the local receiver.
Settings, then Integrations. Each card takes Save token, Save identity and Test connection, then Enable (Sentry and GitHub read Connect and backfill 30 days instead). A token is written to this machine's data directory .env under the variable the card names. Vercel is the deploy anchor rather than an error source, so an armed Vercel on its own leaves the first loop stage at zero. If you would rather arm nothing, press Send a test event on the Incidents tab and copy one of the three pastes the server builds: the curl, the uncaughtException handler, and the CI step that reports what shipped.
- Install the Session-Id hook in the repository you want attributed.
Run npx repoops install inside that checkout. It copies the prepare-commit-msg hook and the pure trailer module in, sets core.hooksPath to .githooks, and prints a wrapper line for every non-Claude agent it detects. Claude Code exports CLAUDE_CODE_SESSION_ID itself; REPOOPS_SESSION_ID takes precedence; REPOOPS_NO_SESSION_TRAILER=1 skips one commit. Without the trailer the band stops at strong, and a squash body naming two different sessions names none.
- Know when the pass runs.
Sentry, Vercel and GitHub pull every 15 minutes, PostHog hourly, the slow-query log every 5 minutes. Correlation itself runs in the nightly pass at 02:00 local, right after the production chains step, over the newest 500 production rows. A post to the local receiver correlates at once, scoped to that event's own timestamp, so trying the tab does not mean waiting until tomorrow. REPOOPS_BRAIN_DREAM=0 stops the nightly pass.
- Read the loop panel before you read the cards.
Seven stages in dependency order, each of which has to be non-zero before the next one can be. Only the first empty stage carries its note, and there is no percentage anywhere, because a loop stopped at stage one is not 14 percent finished. Above the stages, The production pull is not healthy appears only when a cursor is stale (three of its own ticks), erroring, or armed and never finished a pass. A read that failed hides the panel rather than printing zeros.
- Judge the attributions you can see.
Each card asks Is this attribution right? and offers Not my change and Attribution is right. A verdict takes a name, because an anonymous dispute cannot be followed up. The band is read from the server's own row so nobody can move a band's measured accuracy by choosing one. A rate appears on the calibration panel only once five judgements exist for that band, with a 95 percent interval, and a band is flagged below its name only when even the optimistic end of that interval misses what the name claims.
- Open the work on the case, not on the card.
The Attribution page lists the open production and performance cases, overdue first, then by severity, and opens one in the case dialog on its Attribution section: one row per edge with the mechanism, the confidence, the review state and the candidate count, plus Confirm, Dispute and Withdraw per edge. A production incident carries no severity by construction, so the only floor it can clear is recurrence, and the default is three distinct sightings.
- Read Evidence coverage, then retrieve what is missing.
The coverage matrix names each family's state and what it means. On the hosted case, Retrieve missing evidence asks only the installations linked to this case's sessions, and only for the families that can be requested. On the desktop the button is disabled, with the reason: Retrieval runs from the hosted case, where an owner or admin can ask an enrolled installation.
| Setting | Where | A sensible choice | Why it matters |
|---|---|---|---|
sentry.enabled | repos.config.json, per repository; Settings, Integrations writes it | true once the org, the project slug and the token variable are set | Off by default. Sentry is the error source the first loop stage counts, and the stage stays at zero without one. |
sentry.tokenEnv | repos.config.json, per repository | SENTRY_READ_TOKEN | Names the variable the pull reads. The build-time release token is a different token and answers 403 on the issues API. |
vercel.enabled | repos.config.json, per repository | true when you want deploys anchored | Vercel is registered as the deploy anchor, not an error source, so arming it alone does not satisfy stage one. |
dbSlowLog.path and dbSlowLog.driver | repos.config.json, per repository | a local slow-query log, and postgres or libsql | No token: the file is read at pull time. A query shape carries no source file, so these events stay unattributed on purpose. |
VERCEL_LOG_DRAIN_SECRET | the data directory's .env | the drain's own signing secret | Unset, the drain route answers 401 to every batch. Per repository, vercel.logDrainSecretEnv names a different variable. |
REPOOPS_SELF_INGEST | the data directory's .env | leave it unset | RepoOps ingests its own errors by default; 0, off or false turns that off. |
REPOOPS_AIR_PROD_RECURRENCE | the data directory's .env | 3 (the default) | How many distinct sightings a production incident needs before it opens a case. It has no severity, so recurrence is the only floor available to it. |
REPOOPS_EVENTS_RETENTION_DAYS | the data directory's .env | 30 (the default) | How long a production event stays in its day file before it rolls into the gzipped monthly archive, which is still read in range. The incident rows have no equivalent. |
REPOOPS_BRAIN_DREAM | the data directory's .env | leave it unset | 0 stops the nightly pass, which is where the scheduled correlation runs. Posts to the local receiver still correlate. |
What you should see
A chain that earns a lesson
Configuration. Sentry armed, the Session-Id hook installed in the checkout, and a session captured inside the last seven days.
Expect. A card whose band chip reads exact or strong, with the short sha and the author under Change and the session id under Session. Save as lesson is offered, and the guard box opens pre-filled from the culprit line with the directory it was scoped to.
Verify. The loop panel's fourth stage, incidents banded strong or exact (saveable), is above zero. Saving reports Saved or Folded onto the existing lesson, and the lesson appears on the Lessons tab.
An alert with nothing to blame
Configuration. A PostHog metric, a slow-query row, or an error whose payload carries no file and line.
Expect. The band chip reads unattributed and Change reads unattributed. In place of the save control the card reads that the confidence is below strong, because a rule minted from a guess gets enforced forever.
Verify. The row is still counted and still listed, which is the point: the event is kept rather than dropped. The loop panel names the first empty stage and only that one.
A band that is not what its name claims
Configuration. At least five human judgements recorded against one band.
Expect. The calibration panel prints that band's rate with a 95 percent interval. Exact claims at least 95 percent and strong at least 80 percent; weak and none claim nothing, so they are reported without a claim to miss.
Verify. Below five judgements the row reads not measured with the count beside it, never a percentage. A band is marked below its name only when the interval's upper bound falls under the claim, and the panel recommends rather than acts: no write floor moves from there.
Data and cost
- What is captured
- One incident row per attribution, appended to the repository's own incidents/<yyyy-mm>.jsonl: the source, the event kind and id, the timestamp, the message capped at 500 characters, the implicated file and line, the issue URL, the vendor's declared level, the blamed commit sha, author and time, the session or the candidate list, and the band. The persisted causal link sits beside it under causal-links/, and human verdicts under incidents/attribution-verdicts.jsonl. The production events themselves are in the repository's event store, redacted on the way in.
- Who can see it
- The desktop's incident rows, causal links and verdicts stay in the repository's brain directory; this page uploads none of them. The hosted Attribution page shows the team's own cases with their edges and coverage, and captured transcript content reaches the team only through the repository's managed capture policy and an owner's retrieval request, which is audited. One narrow thing can leave, and only when a person turns it on with no desktop control for it: risk publishing sends a per-repository count of open incidents plus at most five labels built from the signal source and kind alone, each run through the egress redactor. Never the message, the stack, the file, the commit or the name.
- How long it is kept
- Production events stay in their day file for 30 days, then roll into a gzipped monthly archive that is still read in range, so nothing is dropped. Cases are read over a 24-month window. The incident rows themselves have no retention: no sweep, no purge route, no knob. That directory grows.
- What leaves the machine
- The correlator makes no model call, and neither does the guard draft: the pattern is a ranked scan with a stoplist, because a model choosing it would put an opinion in the causal chain. The only outbound traffic is each armed adapter pulling its own service on the token you named. The local receiver and the log drain are inbound.
- What it costs
- No metered cost. A pass reads at most 500 of the newest production rows, git-blames one line per event, and skips the rename check entirely when the commit named its session. There is no per-case charge, because there is no call to charge for.
When the result differs
| Symptom | Likely cause | Next action |
|---|---|---|
| The tab is empty and the loop panel shows seven zeros. | No error source is armed. An armed Vercel does not count, because it is registered as the deploy anchor. | Arm Sentry in Settings, Integrations, or press Send a test event and post the curl. Seven zeros is a reading, not a failure. |
| Every row reads unattributed. | The line could not be blamed, or no captured session overlapped it. With no candidate session the band is none and the link is discarded, which reads the same as a blame that failed. | Check that the file still exists at that path, and that a session was captured in the last seven days. A blame failure prints one line to the aggregator's stdout naming the file and line. |
| The band never reaches exact. | The commit carries no Session-Id trailer, or it names two different sessions, or the session it names is not one this machine holds. | Run npx repoops install in that repository and commit again. A recorded graft can recover the trailer through a squash, rebase, amend or cherry-pick. |
| Save as lesson is missing from a card. | The band is below strong, and the panel says so in place of the button. | Nothing to fix. The write path refuses the same floor with a 422, so offering the button would be an invitation to an error. |
| The source is armed and nothing arrives. | The cursor is stale, erroring, or has never completed a pass. Stale means three missed ticks of that adapter's own cadence, not one. | Read the banner above the stages, which names the adapter and how long ago it last pulled, then press Test connection on its card in Settings, Integrations. |
| Retrieve missing evidence is disabled on the hosted case. | The reason sits under the button: the repository has no managed capture policy, the policy is paused, you are not an owner or admin, the team is not on the hosted plan, or the case names no repository. | Act on the reason it names. A case with no linked agent session has no transcript to request; its runtime and GitHub evidence still applies. |
| The request reads Expired. | No result arrived within 72 hours, usually because the linked installation stayed offline. | Bring that installation online and press Retrieve missing evidence again; stored content is checked first, so nothing is asked twice. |
| The stored file holds two rows for one error. | Two passes attributed the same problem differently, and a row's id keys on the signal and the attributed sha together, so the better answer minted a new row rather than replacing the old one. | Nothing to fix. The read folds them to one row per problem and keeps the better-attributed one. The file keeps both on purpose, as the log of what each pass concluded. |
- Disable
- Disable on the source's card stops that pull. REPOOPS_BRAIN_DREAM=0 stops the nightly correlation. Unsetting the drain secret closes the drain. The local receiver stays open, and there is no setting that closes it.
- Roll back
- Not provided. No route re-runs a correlation, un-attributes a row, or edits a band. The appeal is the dispute: it records that the attribution was wrong, with a name and optionally a replacement, and it never rewrites the row it appeals.
- Revoke access
- Press Remove on the source's card to clear the token from this machine's data directory .env. It is not revoked at the service, so revoke it there as well. Unset VERCEL_LOG_DRAIN_SECRET and every further batch is refused with 401.
- Delete
- Not provided. No route deletes an incident row, a causal link or a verdict. The files are incidents/<yyyy-mm>.jsonl, causal-links/<yyyy-mm>.jsonl and incidents/attribution-verdicts.jsonl under the repository's own brain directory. Production events age into the gzipped archive at 30 days; these do not age at all.
Related tasks
Maintenance evidence
- Feature id
incident-correlator(spine leafattribution)- Owner
- The Production Accountability Bridge (PAB.1 to PAB.6) and the Attribution pillar landing tab (P5 slice 9, LDG-0681 step 2, taken inside LDG-0688). Guide: LDG-0717.
- Supported product version
- RepoOps main at 509e8e8ad (after v0.3.2)
- Last verified
- 2026-09-26, read again against origin/main at 509e8e8ad for the edge mechanisms and the Confirmed cause row (public/lib/case-sections.js), the case dialog (lib/ai-incident-case-dialog.mjs), Evidence coverage and retrieval (lib/raw-fidelity/case-coverage.mjs, website/components/case-coverage.tsx, website/lib/raw-fidelity/requests.ts, website/lib/managed-labels.ts), the transcript dialog (website/components/evidence-snapshots.tsx), the hosted page (website/app/team/(home)/agent-traces/page.tsx) and the scope bar (public/lib/scope-bar.js). First read 2026-09-15 at 52366bb6d; labels read from the served tab source (public/incidents.html, public/attribution.html, public/lib/pillar-landing.js, public/lib/case-sections.js, public/incident-case.html, public/settings.html). The band rules, the pull cadences, the retention windows and the seven loop stages were read from the modules, not from the feature doc, and the two disagree in one place: docs/features/incident-correlator.md names a likely band, which bandFromAnalysis never returns.
- Example fixtures
- No fixture file for the chain. The shapes are inline in lib/incident-correlator.test.mjs and lib/incident-correlator.window.test.mjs (extraction, shaping, ranking, the fold, the newest-first read window), lib/causal-link.test.mjs (the band decision), lib/causal/attribution-verdicts.test.mjs (the sample floor and the interval), lib/causal/guard-draft.test.mjs, lib/causal/session-trailer.test.mjs and lib/causal/squash-graft.test.mjs (the trailer and the graft, which are also what exercise lib/causal-correlator.mjs, since it has no test file of its own), lib/routes/incidents-case-join.test.mjs and public/lib/pillar-landing.test.mjs.
- Source references
lib/incident-correlator.mjs,lib/causal-correlator.mjs,lib/causal-link.mjs,lib/causal/session-trailer.mjs,lib/causal/attribution-verdicts.mjs,lib/causal/guard-draft.mjs,lib/routes/incidents.mjs,lib/routes/prod-events.mjs,lib/routes/loop-status.mjs,lib/prod-cursor-health.mjs,lib/brain-cron.mjs,lib/brain-events.mjs,lib/risk-publisher.mjs,public/incidents.html,public/lib/pillar-landing.js,public/lib/case-sections.js,lib/raw-fidelity/case-coverage.mjs,website/components/case-coverage.tsx,website/lib/raw-fidelity/requests.ts,website/lib/managed-labels.ts,website/components/evidence-snapshots.tsx,website/app/team/(home)/agent-traces/page.tsx- Documentation review
- Revised 2026-09-26 by the authoring agent against the code (LDG-0988); independent review not yet recorded.
- Video review
- Narrated story rendered and published 2026-09-26 (render 0a6f1938a100, LDG-0988), beside the desktop navigation with the six pages as they ship. Six frames, the captions and the transcript were reviewed by the authoring agent, not an independent reviewer; the audio was not listened to by a person. It is a labeled synthetic explanation, not a recording of the app.
Last updated