The test passed, but did the agent fix or rewrite?

A repair can satisfy the suite and still modify far more code than the defect required. When agents are free to regenerate an entire function, the green-test rate no longer answers an important question: was the change precise, or did it merely find another route to the same observed result?

Status: Evidence Brief approved for publication. This text analyzes the *Precise Debugging Benchmark* (PDB), a preprint with public code, dataset, and results page. It does not claim that Trustyu Forge reproduced the experiments or that the results automatically transfer to another language, codebase, model, or organization.

The gap the benchmark attempts to measure

Repair benchmarks often ask whether the fixed code passes the tests. PDB adds two measures: edit recall, which checks how much of the reference fix was recovered, and edit precision, which measures how much of what the model changed belongs to the expected fix region.

This distinction matters because an agent can produce a functional solution while changing unrelated lines, functions, or structures. For those who need to review, audit, and operate software, the cost is not only the remaining defect. It is also the radius of change that must be understood.

Population and method

PDB-Single starts with 7,589 examples containing one synthetic defect and retains 5,751 after quality filters. The corpus derives from BigCodeBench and LiveCodeBench, with Python programs and reference fixes.

PDB-Wild brings together 484 larger tasks: 256 derived from BigCodeBench and LiveCodeBench, selected from programs longer than 35 lines, and 228 derived from SWE-smith. For the iterative agent evaluation, the authors randomly sampled 500 instances from the BigCodeBench portion.

The researchers compare unit-test passing, edit recall, and edit precision. Equivalence with the reference fix is approximated through structural matching of normalized lines and the smallest contiguous changed region. This is a useful operational definition, not formal proof that every different edit is unnecessary.

The result that changes how to read a green test

In PDB-Single, some frontier models exceeded a 76% unit-test pass rate but remained below 45% edit precision. In one reported example, Qwen3-Coder reached 70% on tests and 66% precision.

The benchmark also shows that functional performance and precision do not necessarily move together. In iterative rounds, allowing up to three attempts and providing execution or test feedback increased passing and recall, but precision did not improve consistently and often declined. In the slice presented by the authors, Claude Code with execution feedback was around 50% precision.

The point is not that a fix different from the human one is wrong. The point is that green tests and change scope are distinct dimensions. A team may accept an alternative solution, but it needs to see what changed, why it changed, and what additional risk came with it.

Cross-validation and limits

The authors applied the same metrics to 40 single-bug examples from DebugBench. In this slice, GPT-5.1-Codex achieved a 90.0% pass rate and 61.9% precision; Claude Sonnet 4.5, 87.5% and 78.4%; Gemini 2.5 Pro, 85.0% and 79.4%. This is cross-validation conducted by the authors themselves, not an independent external reproduction.

The study is restricted to Python. Much of the corpus uses synthetic defects; edit matching may penalize correct solutions that differ from the reference; and the design does not directly measure long-term maintenance, production incidents, or review cost. The preprint also does not assess how training or post-training data influenced each model.

The authors report affiliations with the University of Southern California, Microsoft, the University of Wisconsin–Madison, and the University of Toronto, with support from National Science Foundation grants. Corporate affiliation does not invalidate results, but it should remain visible. As of the 10/06/2026 cutoff, we found no independent reproduction of the complete benchmark.

How to turn the finding into a gate

1. Test behavior and scope: keep the functional suite, but measure changed lines, files, symbols, and dependencies.
2. Define a change budget: the higher the criticality, the smaller the unjustified edit should be.
3. Ask for intent before the patch: hypothesis, expected region, and anticipated risk make deviations more visible.
4. Compare alternatives: when two fixes pass, prefer the one that produces better evidence — not necessarily the smallest by raw count.
5. Require justification for expansion: a change outside the expected area requires an explanation and proportional review.
6. Rerun relevant tests: textual precision does not replace behavior, security, compatibility, or observability.
7. Record human acceptance: model, prompt, diff, tests, result, and residual risk must remain linked to the commit.

Architecture rule

Coverage and test passing are partial evidence, not a quality verdict. A serious harness keeps behavioral evidence, the changed surface, and the decision that accepts residual risk separate. [API31-C4]

In Forge, the question after a green test should be: which defect was fixed, which region changed, which changes were unnecessary for the original hypothesis, what was verified again, and who authorized promotion.

Direct sources

1. Precise Debugging Benchmark: Is Your Model Debugging or Regenerating? — arXiv. Preprint, method, samples, results, and limitations.
2. PDB — official page and leaderboard. Results and public benchmark description.
3. PDB — official code on GitHub. Implementation of the metrics and evaluation scripts.
4. PDB-Single-Hard — official dataset. Public sample of the corpus.

Editorial and responsibility note

This text combines facts attributed to the sources with the author's analysis and technical proposal. Personal and professional views are not proven facts; data, denominators, limits, affiliations, and the absence of independent reproduction are stated when available. The content is informational and does not replace technical, legal, financial, or security assessment. Tech Human and Trustyu work commercially on related topics. Research, structure, and writing were assisted by AI; factual review, author approval, and publication were authorized by Fernando Parreiras on 10/06/2026, without additional independent human review.

Research cutoff: 10/06/2026. Editorial status: special Evidence Brief authorized for 10/09/2026 at 09h BRT.