Prove · The claim card
The claim card
A benchmark number without its controls is a percentage, not a finding. The claim card is one artifact per run listing the eight controls that could have been applied to it, which of them ran, and the single sentence those fields support. Every control has an explicit value meaning it did not run, so a control nobody applied can never render as a control that passed. The schema is public so anyone can score a published lift against it, including ours.
Where to find it
- On every leaderboard row:
/evals.html?repo=<id>, in the Benchmark Workbench section - Schema:
website/public/schemas/claim-card.schema.json - In the signed bundle: the card rides inside the hash, so a bundle cannot be re-issued with its conditions stripped
- Producer:
lib/benchmark/claim-card.mjs
Why it exists
We built eight controls over five phases: confidence intervals, a paired test, a stated minimum detectable effect, a difficulty screen, a leakage gate, pre-registration, a placebo arm, and a negative control. Every one of them is real, and every one of them was invisible. A row where all eight ran and a row where none did rendered as the same percentage, so the controls bought credibility with nobody.
There is a second reason, and it is the honest one. We benchmark our own product. No amount of statistics fixes that. What can fix it is publishing the controls in a form somebody else can check, and a published bundle they can re-run to disagree with us in public.
The eight controls
| Control | The attack it answers |
|---|---|
| Pre-registration | "You picked the metric that gave you a number." The hypothesis, its falsifier, the single primary metric, the target effect and the look budget are declared before the run, and a run whose configuration contradicts its declaration refuses to start. A declaration dated after the run began records what was found rather than what was predicted, and counts as no declaration. |
| Placebo arm | "The lift is the extra text in the prompt, not the memory." A third arm gets memory of the same shape, the same record count and a matched token budget, drawn from records that are real, adjacent and causally irrelevant. If the placebo reproduces the lift, the run reports no attributable memory effect however large the raw number, and the claim sentence refuses the word memory. |
| Negative control | "Your held-out slice moved too, so it was not memory." Memory of a class the agent has never seen cannot raise accuracy on that class, so a detectable lift there invalidates the recurring number rather than sitting beside it. A novel slice too small to have detected the recurring lift reads indeterminate, not pass. |
| Leakage gate | "The answer was in the memory store." Shingle-overlap matching between the store and the answer keys, audited before the run and over the chunks actually retrieved during it. A case that can read its own answer is excluded and named; a run where nothing clean survives is refused. |
| Difficulty screen | "Half your cases were already solved." Cases the cold agent already answers have no headroom for memory to matter. When the screen ran, the reported lift is an upper bound on the unscreened population and the card says so. |
| Detectable effect | "You found nothing, or you could not have found anything?" Every verdict that declines to call a difference carries the smallest lift the run could have caught at the sample size it had. At an 83% baseline, a with-arm scoring 100% does not clear the intervals until 46 cases per arm. |
| Paired test | "You compared two independent samples that were not independent." Both arms score the same cases, so the comparison is paired. McNemar's exact test on the discordant pairs is reported beside the conservative unpaired reading, and any disagreement between the two is stated rather than resolved in our favour. |
| Independently re-run | "Nobody has ever checked this." A published bundle can be replayed by anyone, and their verdict is recorded next to the original. A disagreement never replaces the original reading: both stand. Zero re-runs reads never re-run, not a neutral blank. |
The rule the card rests on
Every control has an explicit value meaning it did not run: unknown, uncontrolled, not-run, not-preregistered, never-re-run. All of them count as missing in the completeness summary, which reports both a count and the names of what is absent, because the count tells you how much to discount and the names tell you what to ask for next.
This is the field most likely to be quietly wrong, and it was. An early version counted "no placebo arm ran" as a control that had run, because the status string was more specific than "unknown". A run with no placebo therefore reported eight of eight controls applied. The end-to-end test caught it, and the set of did-not-run statuses is now named in one place rather than inferred.
The claim sentence is derived from the fields rather than written beside them, so a headline cannot drift from its evidence. It begins with "No claim" when the leakage gate or the negative control failed, and it never contains the word memory when the placebo reproduced the lift.
Score us with it
The schema is published so it can be pointed at anyone, and a standard its author exempts itself from is marketing. Our own current rows fail several fields: most have no placebo arm, the negative control usually reads indeterminate because the novel slice is small, and no bundle of ours has been independently re-run. Those show as missing on our rows, in the same place, in the same words.
If you publish a lift for a memory or context layer, the useful question is not whether your number is bigger. It is which of the eight ran.