One evening I asked two AI reviewers to check the same deployment plan — an SEO fix for professionalpanorama.com, my 360° panorama photography site, about to go live. The frontier one (OpenAI’s Codex CLI) found eight real defects and changed what I deployed — those eight, confirmed by what I then had to fix in production, became the benchmark, which also makes Codex’s own 8/8 a near-tautology rather than a fair contest score. My local model — a 30B-parameter Qwen on my own RTX 3090 — read the same material and approved the flawed plan with compliments.

That gap annoyed me enough to measure it, and this article is the measurement: a chain of questions, each answered by a graded experiment, each answer raising the next question. The fixed target behind every test is the benchmark: the model receives a prompt — the proposed fix plan + the site’s pre-fix PHP source + the experiment’s instructions — and replies with a review. Each review is scored against the 8 ground-truth defects (all real: every one had to be fixed before deploying): 1.0 per catch, 0.5 per partial, out of 8, plus one binary check — did it reject the one broken fix? Scoring is done by a judge script, grade.py: a keyword scan, then a second local model (Qwen3-Coder-30B, temperature 0) ruling per defect in the form {verdict, quote, reason} with the quote mechanically verified — and a human check on every credited catch, which stayed necessary to the end (more than 150 overrides). Hardware: one RTX 3090, llama.cpp and later vLLM.

It helps to know what the eight defects actually are, because the rest of the article refers to them constantly. Three of them are flaws in the plan itself: it hands one page’s image to another page (the plan-changing one), it adds a route /t/{slug}-lp.jpg that collides with real identifiers ending in -lp, and it points the licence metadata at a page that states no licence terms. Three are bugs in the code the plan builds on: the router compiles route patterns into regular expressions without escaping them, so the literal dot in .jpg matches any character; structured data is emitted with json_encode lacking JSON_HEX_TAG, so a </script> in a title breaks out of the script block; and a query picks one row with LIMIT 1 and no ORDER BY. Two are gaps the plan leaves open: the map feed keeps publishing the old image URLs, and the pages never contain a real <img> of the image they want indexed.

What varies between tests is not only the wording: some use a single request, some chain two, some hand the model tools, and later ones change which model plays which role. What stays fixed is the target — the same plan, the same source, the same eight defects, the same grader. Each test below shows its parameters as one line: model · prompt · seeds · temperature → result. “seeds 1234–1236” means the same prompt was run three times with three random seeds — three different reviews, each scored.

QUESTION 1How bad is the local model at this, really?

Re-run the original review as a controlled test: the plan, code excerpts, a normal polite review request.

Conclusion: reproducibly blind, not unlucky. First suspicion: maybe it simply can’t see enough of the code?

QUESTION 2Does it help to show the model ALL the source code?

Same request, but every PHP file the plan touches — router, controllers, templates — pasted in full: a ~19k-token prompt, most of it source.

Conclusion: context alone does nothing. Maybe passive reading is the problem — what if it explores actively?

QUESTION 3What if the model reads files itself, like an agent?

Instead of pasted code, the request declares three tools — list_files, read_file, and a keyword search over reference docs — over a snapshot of the source tree. A Python loop executes whatever the model calls, up to 12 turns, then asks for the final review.

Conclusion: the trajectories are damning — it opened the buggy router, looked at the buggy line, moved on. The bottleneck is not information access; it’s defect recognition. So: better instructions?

QUESTION 4Does a review checklist help?

Append a generic “check routing, escaping, caching, metadata…” list to the request. No enforcement — just availability, the way a human reviewer keeps a checklist open.

Conclusion: available ≠ applied. What about splitting the thinking into steps?

QUESTION 5Does thinking in two passes help — risk map first, then review?

Two chained requests: the first asks only for a risk map of the plan; the second gets that risk map pasted in and asks for the review.

Conclusion: a blind model writing itself a map is still blind. What if a second model attacks the first one’s work?

QUESTION 6Does a skeptic model attacking the draft help?

Run a normal review, then a second call: “here is a draft review — attack it, find what it missed.”

Conclusion: small models agree by default — a theme that returns later. Five levers, five nulls. What if we stop asking and start ordering?

QUESTION 7What if the prompt ORDERS the model to find defects?

One added contract: “This plan is known to contain defects. A zero-defect approval is a failed review. Argue against every fix before approving it.”

Conclusion: the first lever that moves. Polite prompts make small models validate the plan’s own rationale; adversarial framing licenses criticism. Does it stack with the context levers?

QUESTION 8Does adversarial framing + full code (or tools) stack?

The combinations: full pasted source + the contract; agent tools + the contract.

Conclusion: all the movement comes from the framing; code and tools still add zero by themselves. Can randomness shake more out?

QUESTION 9Does higher temperature find more?

Same combined prompt, temperature raised from 0.3 to 0.7 — more diverse sampling paths.

Conclusion: randomness doesn’t raise the average, but different samples catch different defects — remember this for later. Meanwhile: is the model itself the ceiling?

QUESTION 10Is a different local model better at this?

A tournament — because “which local model is best” is the internet’s favorite fight, usually argued from vibes. Here they meet on a measured task: every model runs the two best configurations (full-code+adversarial, agent+adversarial), byte-identical prompts, only the model changing. The contenders:

• Qwen3-Coder-30B — the coding specialist, and the baseline so far
• Qwen3-32B — general model with thinking mode
• Qwen3-30B-2507 — the newer general Qwen
• gpt-oss-20b — OpenAI’s small open-weights model
• gpt-oss-120b — its big sibling, CPU-expert offload to fit the card
• an “abliterated” (uncensored) variant — as a control

Conclusion: model choice is the second big lever — 0.5 → 3.0 on identical prompts. Notably the 120B reasons beautifully and still won’t commit to a rejection. New champion chosen; can we push its thinking harder?

QUESTION 11Does maximum “reasoning effort” help gpt-oss?

gpt-oss models expose a reasoning-effort setting (low/medium/high). Tested at all three levels, on both serving stacks, with extended context so long thinking has room.

Conclusion: harder thinking ≠ better reviews — use low for structured output, medium otherwise. The model now criticizes and commits… but it plainly lacks the domain expertise. Can we give it that?

QUESTION 12Can we inject expert knowledge by pasting it into the prompt?

Every test so far used only the model’s internal knowledge — whatever settled into its weights during training. But an expert reviewer carries something extra: current, domain-specific best practices. What if we inject those — compressed to their essence and pasted right into the prompt? First attempt: ten numbered routing/URL practices distilled from real documentation (php.net, OWASP), added to the brief — plain rules at this stage, no worked examples yet. No instructions on using them.

Conclusion: the model reads best practices and applies none. What if application is forced?

QUESTION 13What if the model must CHECK each practice, one by one?

Same practices, plus a mandatory audit walk: for every practice, in order — verdict, verbatim evidence quote, mechanism. An empty section is a failed audit.

Conclusion: knowledge converts only under protocol. But pasting handbooks feels crude — wouldn’t a search tool be more elegant?

QUESTION 14What if knowledge sits behind a search tool instead?

Pasting knowledge into the prompt has obvious limits: prompts have finite size, and every new task domain needs its own hand-prepared practice set. Could the model be more independent — keep a whole library of reference documents on disk and search it itself whenever it needs to? The test: the same documents reachable only via a keyword-search tool in agent mode. Three escalating instruction levels, up to an unconditional mandate: “search every document before reviewing; relevance is not yours to judge.”

Conclusion: a model only searches for vocabulary the task already made salient — it can’t query the thing it doesn’t know it doesn’t know. One run even marked the document “relevant: yes” in its notes and never queried it. Push knowledge; never make the model pull it. So: push everything.

QUESTION 15Full handbook — every defect family covered — plus the walk?

The handbook grows to 14 practices spanning all eight defect families (routing, encoding, data selection, metadata coherence, delivery), each with a worked micro-example; audit walk included. Also tried in agent form.

Conclusion: 3.0 → 5.0. The handbook is the third and biggest lever. But one defect still resists everything — and the failures have a curious shape. Can the protocol dig deeper?

QUESTION 16Can the audit walk run inside a single request?

Efficiency attempt: demand the full enumerated walk and the final review in one call.

Conclusion: inline protocols get pencil-whipped. Separate the phases.

QUESTION 17Two requests — full audit first, then the review from it?

Request 1 produces only the enumerated audit; request 2 receives that audit pasted in and writes the review.

Conclusion: now the enumeration executes — and exposes the deepest failure of the whole project: asked to derive what the router’s regex code produces, the model writes the output with the escaping the code doesn’t perform. It derives what code should do, not what it does. Priors overwrite observation. Can that be forced out?

QUESTION 18Can we force derivations to follow the code instead of the prior?

One added grounding rule: “Quote the source line verbatim; apply only it. If your derived regex contains a backslash the quoted code never inserts, the derivation is wrong” — plus one worked example of a correct derivation.

Conclusion: the last resister falls. But the best single run I could produce settles around 5/8 — no single run ever beat it. Different runs catch different subsets, though (remember Question 9)… so stop asking one run to do it.

QUESTION 19Do many seeded runs together cover everything?

Recall Question 9: different seeds make the model give slightly different answers — and we noticed the differing answers catch different defects. Every test so far used just 2–5 seeds; now scale that observation up and union the replies. Note this test changes three things at once — more seeds, two protocols, twenty times the compute — so it measures the package, not seeding alone. The production ensemble: 10 seeds × the rigid audit protocol + 10 seeds × the free agent protocol — the two styles catch complementary defect families (the walk converts mechanical bugs, the agent finds design gaps). All findings extracted and deduplicated by script, each claim stamped with support: how many of the 20 runs said it.

Conclusion: full coverage exists somewhere in the pile — which is not the same as existing in a report. Keep the two apart, because the rest of the article turns on the difference: eight families are touched across twenty runs; what a single assembled document ends up containing is a separate question, answered in Questions 23 and 25. And either way it is buried in noise a human must arbitrate. Can an LLM do the arbitration?

QUESTION 20Can a stronger model verify the claims?

A pipeline: extract every claim, have the 120B model verify each against the full source — confirm what’s proven, reject what isn’t — and write the final review from the confirmed ones.

Conclusion: verification asymmetry — strict verification kills correct-but-hard-to-prove design insights and keeps provable trivia. Verdict framing is the poison. What if the arbiter may organize but never reject?

QUESTION 21What about an editor that ranks but can’t reject?

All ~212 deduplicated claims + the handbook in one call: “merge duplicates, rank Top-15 by impact; uncertainty lowers rank, never deletes.” Tested on three different models.

Conclusion: framing looks like most of the verifier’s failure back in Question 20 — a similar arbitration job scored 1.0 under a reject-if-unproven contract and 6.5 under rank-only. Not a clean A/B: the verifier judged ~100 claims from 8 runs, the editor ~212 from 20, so scale differs too. But the direction is unmistakable, and no other change plausibly turns 1.0 into 6.5. But rare true claims still drown in a 200-claim context (crowding). Shrink the context?

QUESTION 22Consolidate per theme first, then let an LLM assemble the final list?

Claims are split into ~10 topic buckets by a hand-written keyword table; a small call tidies each bucket (crowding is reduced, not eliminated — buckets ran from 2 to 50 claims); a final LLM call assembles the full ranked list from the per-theme results.

Conclusion: every LLM that owns a final inclusion decision loses something. So take the decision away from it.

QUESTION 23What if code assembles and no model can drop anything?

Same per-theme consolidation — then a script concatenates all of it (concatenation cannot drop) and splices in the verdict section from the best rank-only run.

Conclusion: the real defects are now in the final document, produced with no human in the loop — and that is all the score measures: coverage. Be precise about what it doesn’t measure. The document holds 41 findings, so the real ones sit among ~33 plausible-but-false claims; nothing in the metric penalises length, so a longer report can never score worse. Codex’s 8/8 arrived in ten tight findings; this 8/8 arrived in forty-one. Same number, very different object — and deciding which findings are real was still a job for the human reader. One manual step left. Can that, too, be handed to machines?

QUESTION 24Can machines take over that last human step?

This is a test of the human job left over from Question 23: going through the ~40 findings in the final report and separating the real defects from the plausible-sounding noise. Question 20 already showed that asking one model to verify claims destroys good findings — so this time two independent judges rule on each finding, and code compares their answers instead of trusting either.

Per finding, a script choreographs a court: an Advocate argues from the evidence, a Skeptic attacks, one rebuttal each, then three jurors with different lenses (evidence / mechanism / impact) vote. As a control, a lone confirm-if-proven Verifier rules on the same findings.

Measuring this needs a different exam from the 8-defect benchmark: that one measures finding defects, this one measures judging claims. So I took 14 findings from a real audit’s output, hand-verified each — 8 genuinely real, 6 confidently false — and scored the machines on how many of the 14 they classified correctly.

Conclusion: neither judge alone beats the pair — their blind spots are opposites. The jury’s impact lens kills true-but-irrelevant claims that the verifier happily “proves”; the verifier’s quote discipline kills claims the debate talks itself into. So the rule is: where both agree, accept the verdict automatically; where they split, hand that one finding to the human. The last manual step doesn’t disappear — it shrinks from reading forty findings to glancing at the four the machines couldn’t settle. (Two warnings from the transcripts: debaters sometimes swap positions mid-debate, and one skeptic fabricated a supporting quote. Scripts and evidence packs keep them caged.)

One caveat that turns out to matter enormously: those 14 findings were site-audit claims — checkable against observable facts like “is there a canonical tag on this page”. Which raises the obvious question, and the one I should have asked before celebrating anything.

QUESTION 25So — run that filter on the code review and get a clean report?

This is the experiment the whole chain has been walking toward: take the 41-finding document from Question 23, run the two-judge court over it, and report both axes — recall (do the real defects survive?) and precision (what share of the survivors are real?). Same court, same model, same seeds; the evidence pack is the plan and source the finders saw.

It killed the two hardest-won findings in the entire project, each in its own way. The router-regex defect died because the advocate argued against its own side and asserted “all route patterns contain no dot characters” — while the evidence pack plainly lists /sitemap.xml, /api/panos.geojson and /robots.txt. All three jurors and the verifier repeated the fabrication; the evidence juror checked the quoted code snippet and never questioned the unquoted assertion beside it. The slug-collision defect died because a juror wrote that “slug generation via Slugger::slugify() (not shown) removes such characters, so the routes remain unambiguous” — flagging that the code was absent from the pack and reasoning from it anyway. A second juror went further, asserting that slugify “removes periods and other non-alphanumeric characters, ensuring slugs never contain . or -lp“, which is exactly backwards: slugify produces hyphens.

One kill, though, was right — and it cost me the headline. Both surviving claims about non-deterministic ordering pointed at a query that demonstrably does have an ORDER BY. The real defect is a different subquery, and no finding among the 41 located it. The judges caught a generosity my own human confirmation had missed: the honest recall of that document is 7 of 8, not 8 of 8.

Conclusion: the filter buys precision at a terrible exchange rate on this material, and both bad kills share one cause — the court had no anti-prior rule, the very discipline the finders needed to catch those defects in the first place. So I ported it in and re-ran.

QUESTION 26Does grounding the judges fix it?

Every role gets the Question 18 treatment: claims about code need verbatim quotes, absence of code from the pack is never proof that behaviour is safe, arguing your assigned side is mandatory (say “concede” rather than switching), the evidence juror may not back an unquoted assertion, and the verifier gains an “unproven” verdict that cannot kill a finding.

Half a rescue, and the half that failed is the instructive one. The regex defect survived — but not because the jury got it right: it voted 3–0 to kill it again, this time citing three genuinely dotless routes from a list that also contains the dotted ones. It lived only because the verifier answered “unproven” and unproven can no longer kill. That is a failure mode quote-checking cannot catch by construction: selective quotation — real quotes supporting a false universal claim.

The slug-collision defect died a third time, by the exact error Question 18 was invented to fix: the skeptic quoted the router’s compiler correctly and then wrote the derived regex with an escape the quoted code never inserts, and the other jurors copied the derivation. My judge-side rule was abstract (“apply only what the quoted lines do”); the finder-side rule that worked was mechanical (“a backslash the code never inserts means your derivation is wrong”), with a worked example. Generic grounding gets pencil-whipped — the same lesson as the retrieval mandates and the single-call protocol, now recurring in a third place.

Conclusion: the grounded court is safer and almost useless — it defers 39 of 41 findings, so the precision gain evaporates. Judging a mechanical-derivation defect demands the same concrete, per-family protocol that finding it demanded, at which point the judge is simply re-running the finder. No automated filter I have can clean up that document: aggressive judging costs the crown jewels, grounded judging abstains. The honest end state is 7/8 recall spread across 41 findings, with a human still doing the last separation — for this domain. Where the claims are observable facts rather than derivations, the same court scores 12/14. The difference isn’t the judges. It’s what the claims are made of.

What the chain adds up to

Three levers move a local model: adversarial framing, model choice (with reasoning-effort discipline), and pushed, protocol-forced knowledge. Everything else that “should” help — context, tools, checklists, model-initiated search, harder thinking — measured zero on this benchmark. Past the ~5/8 single-run ceiling, the gains come from structure: seeded ensembles for coverage, and mechanical merging so nothing is lost. Cost of the full system: about 10 minutes of GPU time, $0.

And the honest scoreboard, stated as a pair rather than a single flattering number: recall 7 of 8 real defects, spread across 41 findings of which roughly four in five are noise. That is genuinely frontier-level coverage from a 20B model on one desktop GPU — and it is not the same thing as a good review. Automating the last step, separating the real findings from the plausible ones, is where the chain currently ends: two judges disagreeing gets you most of the way on observable facts, and nowhere at all on defects that must be derived from code.

The one-line conclusion: at local scale, the expertise lives in the harness, not the model. The model supplies cheap, tireless application of knowledge you curated, inside a structure you control — and every time an LLM was given a control decision instead of a generation task, quality fell. Build the structure.

The corollary, learned the hard way in the last two questions: measure both axes or you will fool yourself. A recall-only score cannot go down when a report gets longer, so it quietly rewards volume. I spent a day happy with an 8/8 that was really a 7/8 buried in forty-one findings — and the thing that caught it was not a better model but a stricter question.

Where this went next: I welded the whole recipe into a one-command “black box” — point it at a URL, ten minutes later a debate-filtered audit report comes back, nobody home — then benchmarked it against Codex and a Claude agent on two live websites. It held its own, missed things in instructive ways, got upgraded, and caught one real defect both frontier reviewers walked past. That’s the next article.

What this does not prove

Before the list, the honest frame: this is a few days of one engineer poking at one benchmark on one desktop GPU, not a study. Several of the questions above are, in the real world, entire research programmes — a properly staffed lab would spend a year on any one of them with a dozen people, many benchmarks and statistics I do not have. “Does adversarial framing generalise across task types?” is a paper. “How does defect recognition scale with parameter count?” is a paper. “Can judge ensembles be made reliable on derived claims?” is several. What I have instead is a hundred-odd graded runs on a single frozen case, each conclusion resting on a handful of samples, with me as the only reviewer of my own results.

So read the strong claims as strong on this bench. Some of them — that context alone does nothing, that framing is the master lever, that code should own inclusion decisions — showed up so consistently, across models and formats, that I would bet on them transferring. Others are genuinely debatable and I would not argue hard for them: the exact ranking of the six models, whether the observed 5/8 single-run limit is a real ceiling or just where my patience ran out, whether the two-judge court would behave differently with better evidence packs, and whether any of this survives contact with a codebase that is not mine. If you disagree with a conclusion here, you are probably disagreeing with an n of three, and you may well be right.

• One benchmark, adaptively tuned. Every number here comes from a single frozen review. Worse for generalisation: the handbook was deliberately extended until it covered all eight known defect families, and the pipeline was tuned against the same eight-defect score it is measured by. That is honest engineering for building a tool and terrible epistemics for claiming general capability. Treat “frontier-level coverage” as “frontier-level coverage on the case it was tuned against” until a second frozen case says otherwise.
• Small samples. The lever tests are 2–5 samples per configuration. That is enough to distinguish “moved the needle” from “did nothing” — a 0.0/0.0/0.0 against a 0.5/1.0/1.0 — and nowhere near enough to rank two close configurations. Where I say something “does nothing”, read “produced no median improvement on this benchmark at these seeds”.
• Codex is the answer key, not a contestant. The eight ground-truth defects were defined as the ones Codex found and production then confirmed. Its “8/8” is therefore definitional, and the grader is calibrated to reproduce it. Read every comparison as “how much of what the frontier reviewer found did the local system also find”, never as a fair head-to-head. The setups were unequal too: Codex ran agentically with live web search and ~100k tokens; the local runs read a 20k-token brief on one machine.
• The retrieval result is narrower than it sounds. What failed was model-initiated search: the model would not query for what it did not know it needed. Retrieval driven by code — where the harness picks the passages — does work, it just lost to a hand-distilled handbook in a separate test.
• Judge blind spots are documented, not eliminated. More than 150 verdicts needed human override, in both directions. Every score in this article is post-confirmation, which is exactly why I do not trust the pre-confirmation ones.

Method note: the experiments were orchestrated with Claude Code driving the harness — building the graders, running the ladders, and getting adversarially benchmarked against Codex CLI along the way. All scores are human-confirmed, not raw judge output. Test numbers reference the full parameter log (TESTS.csv).