Eval engineering, and what we shipped

Ten days ago we wrote that after graph comes eval: once the model is a commodity and the verifier is the bottleneck, the verifier stops being a throwaway command and becomes an asset you version and track. That was a prediction. This is the build.

The idea, in one line

The model is a rental. The examiner is yours, because it encodes your own definition of correct. An "Eval Engineering" essay in July put it that way, and a verified research pass on the state of the art backed the mechanism, not just the slogan. Eval vendors have unbundled their suites into installable coding-agent skills, so the bar now travels with the repo.

We already shipped the base of this: a versioned eval-bar, a case set plus a pass threshold, versioned so raising the bar is a recorded event and not a silent edit. The next question was narrower. Which specific mechanisms from the outside world are worth adopting, and which cross a line we drew on purpose?

What we built

Six PRs, each on the same eval-bar substrate, each staying advisory and each refusing to run your loop for you.

that sounds right but was built on an empty tool result. A deterministic tool-parameter-accuracy scorer checks the agent called the right tool with the right arguments against a known answer. Both are eval-bar case dimensions, and a null score fails closed to not-evaluated, never a fabricated pass.

scaffolds an eval-bar case, with the expected outcome held out of the target's view.

every verdict so a silent examiner upgrade cannot make before and after incomparable, and a lint that rejects a rubric scoring length, keyword count, or citations instead of correctness.

trajectory, historical revert rate, and sandbox outcome, with honest-null denominators. It names the failing signal for a human. It does not merge code.

ecosystem standardized on, so an eval authored by the LangChain skill or AWS Agent-EvalKit imports, and a RepoOps eval-bar exports, with a round-trip test proving no loss.

step, the whole trajectory, or a retrieval. Grade only the final answer when the case should grade the path, and you hide a failure reached the wrong way. A trajectory case now says the whole path is in scope.

• Grounding and tool-accuracy judges. A faithfulness score catches the answer
• Trace to eval. A skill mines a recorded failure out of a span tree and
• Judge hygiene. A cross-family judge, the judge model and version pinned on
• Advisory merge-confidence. One score fuses guardrails, recent eval
• Harbor interop. RepoOps reads and writes the Harbor task folder the
• Dataset types. A case can declare what it grades: the final response, one

Two lines we did not cross

We do not auto-merge code with no human, and we do not build a loop runner or a graph execution engine. RepoOps stays advisory: it observes, remembers, verifies, and prices the runs. The merge-confidence score is the clearest test of that line. monday.com uses a composite score like ours to merge roughly nineteen of twenty agent PRs with no human. We compute the same composite and stop one step short: we tell you which signal is failing and let you decide.

What we are keeping honest about

The outside validation is real, and it is uneven, so the comparison page and the plan say so out loud.

Confirmed against primary sources: the LangChain eval-engineering skill and the LangSmith skills are real and installable; AWS Agent-EvalKit is Apache 2.0 with a faithfulness metric; Harbor is the task format the ecosystem settled on; monday.com does gate autonomous merges on a composite score.

Directional only, and labeled that way wherever we cite it:

single-vendor result, and the source itself concedes a rubric change cannot raise a model's capability.

expert-labeled failures. The durable read is the best-case recall near eighty-seven percent, not the exact ordering of vendors.

a self-benchmark of undisclosed size.

• The monday.com claim that changing only the evals moved every score is a
• The blind-test judge ranking (Parlance Labs) is one dataset of thirty-nine
• The LangChain "eighty-two percent with the skills versus nine percent without" is

We would rather ship the mechanism and state the caveat than borrow a number we cannot stand behind.

Where to see it

The Evals tab shows the bars, the pass rate over time, which loops clear which bar, and now the dataset type each case grades. The new Eval engineering tab on Why RepoOps lays the vendors out row by row, each cell citing its own source, framed against what RepoOps actually built.

RepoOps proves you did the work, and remembers why. Eval engineering is how we prove the work was good, on a bar you can version, share, and hold a run against.

repoops.ai