The eval ran. Is the new version actually better?
Paste the export of one eval run - inputs, expected answers, the model's outputs and, if you have them, a baseline's outputs, judge scores, human ratings, slices or retrieved ids. Your browser scores every row, puts a 95% interval on every rate, pairs the candidate against the baseline (wins, regressions, an exact sign test), measures kappa between raters or between the judge and a human, and checks the claim and your ship bar against the rows. Free, before you sign in, and nothing is uploaded. Then the verdict lane says ship, hold or fix the eval first, and the judge lane designs the LLM-as-judge that grades this feature from now on.
The products and rows are invented. Two are verdicts (the summary one also carries its judge design, one click after the verdict) and one is a judge design; each comes with a saved model reply, so you can see the whole page without signing in or spending a credit.
What this does, and what it does not
Most prompt changes ship on one number: the new version scored a few points higher on the eval set. The skill behind this page asks the questions that number hides. Is the gain bigger than the noise of a small set? Which cases got worse while the average went up? Does one kind of request fail while the total passes? Does the judge that produced the score agree with a human at all? All of that can be computed from the rows, so the page does it for free - with intervals, an exact sign test on the discordant rows, Cohen's kappa and a slice breakdown - and checks any claim and ship bar you give it.
The verdict lane reads that scoring like a reviewer: it answers every finding, groups the failing rows into failure modes, lists the regressions, names what the set does not cover, and says ship, hold, or fix the eval first. The judge lane designs the LLM-as-judge for the feature - mode, weighted criteria, the complete judge prompt, calibration rows from your run that a correct judge must grade right, new cases and ship bars. Every id and figure in a reply is checked against your rows.
The paid lanes see up to 48 of your rows (every regression first, then failures, long cells cut and marked) and the page's scoring of all of them. Your paste stays in this browser unless you run a lane. The page measures the run you give it; it cannot tell whether your references are right or your set looks like production. Derived from the agent skill @wshobson/llm-evaluation (wshobson/agents, MIT: automated metrics for classification, text and retrieval, human evaluation with inter-rater agreement, LLM-as-judge in pointwise, pairwise and reference-based forms, A/B testing with statistical tests, and regression detection). Sources are listed in llms.txt.