Assay
An evaluation harness in Python — build the benchmark out of a repository's own git history, then measure which AI coding tool actually works on it.
LEVEL 1THE PROBLEM
Nobody evaluating an AI coding tool for their company has ever asked how it scored on SWE-bench. They ask whether it will work on their monorepo — their internal framework, their bespoke build, their fifteen years of history — and there is no public answer, because their code cannot be shipped to a leaderboard and no public benchmark samples a stack like theirs. Public numbers are also contaminated for any repository older than a model's training cutoff; a private one is not, which makes an in-house evaluation more trustworthy than a public score, not less. Today that question gets answered by feel, over a two-week trial, from anecdotes. Assay turns it into a measurement. It is the second half of Portcall: that one answers can this tool run here, this one answers is it any good here, and those are the two questions of every enterprise proof-of-concept, in that order.
LEVEL 2THE BUILD
A Python harness that mines the benchmark out of the repository instead of authoring it. A merged commit that changes source and tests together is a ground-truthed task for free, in the codebase's own idiom: check out the parent, apply only the test changes, prove the tests fail, apply the real diff, prove they pass. Anything that does not survive that red-to-green gate is discarded and counted, and the yield is published — 1,847 commits examined, 213 tasks kept is a more persuasive number than the task count alone. Trials run in a sandbox with networking off apart from an allowlisted model endpoint, because otherwise a tool will `pip install` its way to a passing test, and one will. Every task runs n times per tool, since these systems are nondeterministic. The report leads with pass^n — all n trials passed — rather than pass@1, because a buyer rolling a tool out to 300 engineers cannot depend on something that works one time in five. Wilson intervals on every proportion, a paired significance test across tools on identical tasks, and cost per solved task on the front page.
BOSS FIGHTTHE RESULT
On the bench: M0, M1 and M2 have closed, and M3 is underway. `assay mine` builds a suite out of a repository's own history — it walks the commits, splits each one into its test half and its source half, applies the test changes alone and demands they fail, then applies the real diff and demands they pass twice rather than once, because a test that only sometimes passes is not ground truth. `assay validate` re-runs that same gate against an existing suite and fails if a task has stopped revalidating, comparing the recorded test sets in both directions so a green run that exercised something else does not count. M2 built the two pieces a run stands on. Every trial now happens inside a per-task container with memory, CPU and process ceilings and no network interface at all — capabilities dropped, no new privileges — and the tests assert those negatives rather than describe them, so a trial that reaches for the network has to fail. Scoring became executable rather than asserted: a trial is decided on its test report and nothing else, and the two oracles are now measured instead of claimed — the ground-truth adapter scores 1.0 and the null adapter 0.0 over every task mined from the fixture repository. Pinning the environment in the task image also paid an older debt — `no_tests_executed` is split out of `still_red`, so 'the fix did not work' and 'no test ran' have stopped sharing a tally. The run against a real repository is still the honest measure of where this stands, and it did not go the way the plan wanted: M1 walked 743 commits of httpie for 0 tasks kept, M2's pinned per-task images re-walked the same 743, and it is still 743 examined and 0 kept. The pinned image did not lift the reach limit M1 recorded, and that negative result is the finding rather than a footnote. What moved is where the failure is counted — 125 of those commits came back as an environment no image could be built for, which is a sentence about Assay rather than about httpie, so it is reported on its own and never folded into the rejection reasons. The constraint from M0 is still enforced in code rather than promised in a README: when two tools' confidence intervals overlap, the renderers suppress any winner claim and say so. M3 is the public `run` command — the end-to-end run, n trials per task per tool, pass^n leading pass@1, Wilson intervals on every proportion, and a naive baseline adapter in every report so a tool's score always has something to beat. The statistics are M4, published results for two tools are M5. Everything below that line is a plan, and this page will say so until it is not.
HIGH SCORES
- pass^nHEADLINE METRICALL N TRIALS · NOT BEST-OF-N
- 4SCORING TIERSNEVER BLENDED INTO ONE NUMBER
- 647TEST FUNCTIONSPYTEST · COUNTED IN CI
- STRICTTYPE CHECKMYPY --STRICT · CI GREEN
- 743 / 0HTTPIE RE-MINECOMMITS EXAMINED / TASKS KEPT
- 58ADRSCOUNTED IN CI