DZEKA

PITCH AN IDEA

Got something worth a cabinet? Pitch it here. I read every one.

PROCESSING CREDIT…
CREDIT ACCEPTED
idea launched — I'll ping you soon
+1 CREDIT · PLAYER TWO JOINED
← BACK TO THE FLOOR
SYSTEM 02 · MISSION BRIEFING

Assay

An evaluation harness in Python — build the benchmark out of a repository's own git history, then measure which AI coding tool actually works on it.

ROLEDESIGN → BUILD → OPERATE
TIMELINE2026 — PRESENT
STACKPYTHON · GIT · CONTAINERS · STATISTICS
STATUSSHIPPED · M5 / M5 ↗

LEVEL 1THE PROBLEM

Nobody evaluating an AI coding tool for their company has ever asked how it scored on SWE-bench. They ask whether it will work on their monorepo — their internal framework, their bespoke build, their fifteen years of history — and there is no public answer, because their code cannot be shipped to a leaderboard and no public benchmark samples a stack like theirs. Public numbers are also contaminated for any repository older than a model's training cutoff; a private one is not, which makes an in-house evaluation more trustworthy than a public score, not less. Today that question gets answered by feel, over a two-week trial, from anecdotes. Assay turns it into a measurement. It is the second half of Portcall: that one answers can this tool run here, this one answers is it any good here, and those are the two questions of every enterprise proof-of-concept, in that order.

LEVEL 2THE BUILD

A Python harness that mines the benchmark out of the repository instead of authoring it. A merged commit that changes source and tests together is a ground-truthed task for free, in the codebase's own idiom: check out the parent, apply only the test changes, prove the tests fail, apply the real diff, prove they pass. Anything that does not survive that red-to-green gate is discarded and counted, and the yield is published — commits examined against tasks kept — because the rate a repository gets rejected at says more about a suite than its task count does. Trials run in a sandbox with no network interface at all — a model is called from outside it, never from inside — because otherwise a tool will `pip install` its way to a passing test, and one will. Every task runs n times per tool, since these systems are nondeterministic. The report leads with pass^n — all n trials passed — rather than pass@1, because a buyer rolling a tool out to 300 engineers cannot depend on something that works one time in five. A band on every rate by the method that metric requires, a paired significance test across tools on identical tasks, and cost per solved task on the front page.

Assay system architecture On the customer's own machine, assay mines candidate tasks out of a repository's git history, puts each one through a red-to-green gate that discards everything it cannot prove, and writes what survives into a content-addressed suite. Each task is then run n times per candidate tool inside a sandbox with no network interface at all, the model being called by the adapter from outside it through an allowlisted endpoint, with a naive baseline adapter kept in every report beside the agentic tool. Attempts land in a results store, which is scored on executable signal and on cost priced at report time from supplied prices, given two bands computed by two methods — Wilson on pass^n, a bootstrap on pass@1 — and a paired McNemar test, and rendered as a redacted report. The repository itself never leaves the machine. ASSAY Python 3.12 · the repository never leaves the machine their repo git history mine source + test commits validate red → green yield reported suite content-addressed versioned n trials per tool sandbox · network off n trials task image tests decide pass or fail patch baseline adapter one model call kept in every report tool adapter agentic CLI driven headlessly allowlisted model endpoint tokens metered results attempt · diff cost · latency score tests pass no regression stats boot · Wilson pass@1 · pass^n report json · html · text no winner declared redaction paths hashed assay module on disk off the machine sandbox their machine
Terminal capture of assay report over a real run: qwen2.5-coder:7b through the naive-local baseline, served locally by Ollama, against the null and ground-truth oracles. Coverage reads 11 of 13 tasks measured. The tool rows show 55 trials each, 165 in all, with pass@1 1.000 for ground-truth and 0.000 for both null and naive-local, which passed 0 of its 55 trials. Its pass^n was 0.000 as well, level with null, so null vs naive-local reads No winner, while ground-truth is named the winner over naive-local. The model ran locally and nothing was spent.

BOSS FIGHTTHE RESULT

Shipped: every milestone from M0 to M5 landed. `assay mine` builds a suite out of a repository's own history — it walks the commits, splits each one into its test half and its source half, applies the test changes alone and demands they fail, then applies the real diff and demands they pass twice rather than once, because a test that only sometimes passes is not ground truth; `assay validate` re-runs that gate against an existing suite and fails if a task has stopped revalidating. M2 put every trial inside a per-task container with memory, CPU and process ceilings and no network interface at all, and made scoring executable: a trial is decided on its test report and nothing else. M3 shipped the public `run` command, and its first run — 2 tasks × 2 adapters × 5 trials, over a suite mined from the fixture repository — had two oracles measure pass^n 1.000 and 0.000 with intervals that still overlapped, so the harness refused to name a winner between two adapters whose answer it already knew. M4 was the statistics. pass@1 is a mean of per-task rates rather than a proportion, so it got a seeded percentile bootstrap while pass^n kept its Wilson band; an exact two-sided McNemar over tasks answers whether two tools differ and is never allowed to rank them — ranking stays the intervals' job alone; and cost per solved task is computed from prices supplied with the report, so the harness carries no price list it cannot keep current. M5 was the public release: redaction made total and applied before anything is formatted, a self-contained HTML report beside the text and JSON ones, and a one-command demo that mines, validates, runs and reports — over the synthetic fixture repository, which it says before the first step rather than after the numbers. Its two published results are the ground-truth and null oracles, named as oracles above the first figure. M5 also pointed the shipped miner at three public libraries and got the project's first non-zero yield on code it was not built against, 600 commits examined and 14 tasks kept. The three were chosen in advance to fit what the miner already reaches, so that is a reach limit sidestepped rather than lifted, and those 14 passed the gate once and have never been re-validated. httpie, walked twice, is still 743 examined and 0 kept. Then, after M5 and owned by no milestone, a model was called for the first time: qwen2.5-coder:7b, served locally by Ollama, one call per trial and no agent loop, over the 13-task suite one of those libraries yielded. Two task images failed to build, and the report names them rather than dropping them, so 11 of the 13 were measured, for 165 trials. The model returned a patch every time and passed 0 of 55: pass@1 0.000, pass^n 0.000, sharing the null oracle's interval exactly, so the report declined to name a winner between them — the first time figures a model produced have reached that refusal. That zero is published as it came out. It is a baseline, not a tool, and no tool has been scored yet; no API key was read and nothing was spent.

HIGH SCORES

  • pass^nHEADLINE METRICALL N TRIALS · NOT BEST-OF-N
  • 4SCORING TIERSNEVER BLENDED INTO ONE NUMBER
  • 1185TEST FUNCTIONSPYTEST · COLLECTED AT b0805d0
  • 0 / 55FIRST MODEL RUNQWEN2.5-CODER:7B · LOCAL
  • 600 / 14PUBLIC-REPO YIELDCOMMITS / TASKS · REPOS CHOSEN TO FIT
  • 73ADRSDECISIONS RECORDED
NEXT LEVEL
Bosun