The PoC passed. Did the patch follow maintainer intent?

A repair can stop the known exploit, preserve the regression suite, and still conflict with the design a maintainer would accept. When a benchmark measures only whether the Proof of Concept stopped working, symptom suppression may receive the same credit as fixing the cause.

Status: Evidence Brief approved for publication. Editorial series: Papers we read for you. This text analyzes the preprint *On the Reliability of LLM-Based Vulnerability Patching Benchmarks*. It does not claim that Trustyu Forge reproduced the experiments, that developer tests are a perfect oracle, or that the percentages transfer to another tool, model, or codebase.

The evaluated dataset

The researchers curated 112 historical security bugs from 84 open-source projects written in C/C++, Go, and Rust. Each case includes three evaluation layers: a PoC tied to the issue, existing regression tests, and additional tests derived from developer-accepted changes. The final layer seeks to capture API decisions, fix location, and design intent that merely stopping the crash may not preserve.

The study ran three controlled experiments and five case studies. It also examined 13 existing benchmarks and vulnerability datasets to identify agent, framework, and dataset factors that change the score without necessarily improving the patch.

The difference between passing the PoC and satisfying the maintainer

Using the same OpenCode harness and 112 cases, the authors compared Claude Sonnet 4.0, Sonnet 4.5, and Opus 4.7. Reported PoC pass rates rose from 77.7% to 84.8% and 90.2%. Developer-test pass rates rose from 43.8% to 48.2% and 60.7%. Regression pass rates were 97.3%, 94.6%, and 94.6%.

Even in the strongest condition, 32.7% of cases that passed the PoC failed developer tests. For Sonnet 4.0 and 4.5, the proportions were 43.7% and 43.2%. This does not mean that one third of all AI-generated patches are wrong. It means that, in that dataset and harness, a PoC-only oracle counted as successes changes that another requirement set rejected.

The harness also changes the score

In the Sonnet 4.0 experiment, providing detailed build and test instructions raised the overall PoC pass rate from 74.1% to 77.7%, while developer-test passes rose from 41.1% to 43.8%. In C/C++, the PoC effect was much larger: 72% without instructions and 92% with instructions. Developer-test passes changed from 28% to 36%.

This does not make instructions bad; an agent needs to know how to operate the project. The point is methodological: scores are not comparable unless they state which information, tools, network access, Git history, and feedback the harness made available. The benchmark measures the model–agent–harness–dataset–oracle combination, not an isolated model capability.

A minimum patch-evaluation contract

1. PoC: proves that the known path no longer produces the exploitable effect.
2. Regression: protects behavior already covered before the fix.
3. Maintainer intent: adds tests for expected semantics, API, location, and strategy.
4. Adversarial variations: changes inputs and paths to distinguish a general fix from blocking a single case.
5. Isolation: removes history, reference patch, network, and artifacts that reveal the answer, except when those resources are part of the evaluated question.
6. Pipeline errors: separates timeout, permission, test collision, and infrastructure failure from an actual patch failure.
7. Human review: preserves judgment proportional to risk before promotion or release.

Architecture rule

A repair receipt cannot be only “green PoC.” It must record the bug, causal hypothesis, patch, information supplied to the agent, executed oracles, side effects, outcome of maintainer-added tests, and who accepted the change. Without that chain, the score measures a lab configuration and does not prove release readiness. [R2-C3]

In Forge, the test suite is a layered evidence contract with provenance. Generating the patch and generating or selecting the oracle are responsibilities that must remain distinguishable; when the same trajectory controls both without review, it may optimize for the proof instead of fixing the product.

Limits and conflicts of interest

The paper is at v1 dated October 7, 2026. The dataset is curated from public historical bugs and may therefore be affected by contamination or memorization. Additional upstream-derived tests are a proxy for intent, not a complete specification or formal proof of correctness. Results depend on the harness, prompts, tools, model versions, and timeout policies used.

By the editorial cutoff, we had not found a complete independent replication or a single public artifact containing the entire announced dataset. The authors report affiliations with Northwestern University; we found no declared commercial conflict with the cited model providers. No identified conflict is not proof of absolute independence.

Direct sources

1. On the Reliability of LLM-Based Vulnerability Patching Benchmarks — arXiv. Preprint, authors, abstract, version, and date.
2. On the Reliability of LLM-Based Vulnerability Patching Benchmarks — full text. Method, denominators, results, case studies, survey, and limitations.

Editorial and responsibility note

This text combines source-attributed facts with the author's analysis and technical proposal. Personal and professional views are not established facts; data, denominators, limitations, affiliations, and the absence of independent replication are stated when available. This content is informational and does not replace technical, legal, financial, or security assessment. Product names belong to their respective owners; there is no sponsorship or endorsement. Tech Human and Trustyu operate commercially in digital-product development and governance. Research, structure, and writing used AI assistance; factual review, author approval, and publication were authorized by Fernando Parreiras on October 8, 2026, without additional independent human review.

Research cutoff: October 8, 2026. Editorial status: special Evidence Brief authorized for October 15, 2026 at 9:00 AM BRT.