Spec-driven without self-deception: coverage, contract, eval and OpenAPI are not the same proof

Teams that adopt AI quickly learn to ask for evidence before accepting a change. The problem begins when disparate evidence is compressed into a single word: “green.” The suite has passed, coverage has gone up, the contract exists, the schema is valid and eval has improved. Therefore, it is concluded that the system is correct.

This conclusion does not follow the premises. Each mechanism answers a different question. When we treat proximity in the pipeline as semantic equivalence, we build a beautiful dashboard for a certainty that was never produced.

Spec-driven does not mean writing a specification and relying on it. It means maintaining a traceable chain between intent, properties, verification mechanisms, evidence and acceptance decision.

Executive summary

A description OpenAPI tells how consumers and tools can understand an HTTP interface; a compatibility diff looks for structural changes; a test run observes behavior; a coverage metric tells you what was exercised; and an eval measures a behavioral property under a set of tasks and graders. They are complementary layers, not substitutes. [API31-C1]

Even within a single layer there are limits. oasdiff documents OpenAPI 3.1 comparison and breaking change rules, but a green diff does not certify business compatibility. Coverage percentage, in turn, does not prove relevance of tests, correction or risk coverage. [API31-C2] [API31-C4]

Responsible design starts with a simple question: Which property does this green really authorize to assert? The answer must be explicit before the pipeline runs.

Five mechanisms, five questions

The table does not create a fixed hierarchy. In a pure library, contract and unit testing may be the center. In an agent that changes data, policy, sandbox, eval and observability may be more important. The composition depends on the risk.

Start with the property, not the tool

“Add eval” is not a verifiable requirement. “In one hundred representative scenarios, the agent never sends a message without valid consent and every blocked attempt generates attributable evidence” is a candidate property.

For each property, record:

1. subject — which version of the system is being examined;
2. stimulus — task, input and environment;
3. oracle — who or what decides the outcome;
4. limit — which condition needs to be satisfied;
5. evidence — where the immutable result can be inspected;
6. authority — which decision the green can release.

Without subject, a report may belong to the wrong commit. Without an oracle, the output becomes opinion. Without limit, any number can be called an improvement. Without authority, green exists but no one knows what it unlocks.

OpenAPI is description; compatibility is a bigger decision

The OpenAPI specification defines a language-independent interface description for people and computers to understand a service's capabilities. This reduces ambiguity in the integration, but does not carry all the business semantics.

Consider a field limite_aprovado which remains numeric and mandatory. The API can remain structurally compatible while a unit, policy, or rounding change breaks the operation. The structural diff deserves to be green; the business invariant test deserves to fail. They are coherent results because they examine different properties.

The error is not in the tool. It’s about presenting “no breaking change” as “no impact”.

Coverage is a map, not a verdict

Coverage shows areas visited during the run. It helps locate silence: branches never exercised, forgotten modules, untested deltas. But an executed line may be poorly checked; a test may reproduce the implementation rather than the requirement; a critical case may represent a small fraction of the code.

In FORGE, useful reading separates at least line, branch, scope, delta and threshold. Still, these fields are signs of testing, not resolved risk. [API31-C4]

To not turn percentage into theater:

• link critical scenarios to the properties they protect;
• use coverage to discover gaps, not to declare correctness;
• treat new code and high risk differently than historically;
• require evidence of the test that fails when property is violated.

Eval is a versioned experiment

An eval needs a task bank, environment, system configuration, grader, aggregation rule and baseline. Changing any of these elements changes the experiment.

An aggregate number can hide classes of failure. An agent can improve the average and worsen irreversible cases. A model grader may prefer style and miss facts. A benchmark may contain broken tasks. Therefore, eval is not a medal; it is a decision infrastructure.

The minimal panel records:

The most important test: violating property

A gate only demonstrates strength when the prohibited condition causes it to fail. Before trusting the pipeline, perform a deliberate negative case:

• remove a required field and confirm contract failure;
• introduce an incompatible change and commit the diff;
• delete the test from a critical branch and observe coverage;
• make the agent try to overcome an authority and confirm the block;
• change the subject of the evidence and confirm that the binding fails.

The negative case distinguishes executable mechanism from aspirational documentation.

An evidence matrix for PR

The matrix does not need to cover each product line. It needs to cover the properties that support the decision being made.

What this article proves — and what it doesn't prove

FORGE sources and contracts support the separation of description, diff, execution, coverage, and eval. They do not define a universal recipe for tools, thresholds or number of tests. A regulated product and an internal prototype need different profiles.

There is also no automatic independence because a second model evaluated the first. If both share context, incentive, error, or authority, there is separation of calls, not necessarily separation of control.

Conclusion

Spec-driven is a traceability discipline, not a label.

A specification declares intent. A contract structures an interface. A test looks at examples. Coverage reveals exercised areas. An eval measures behavior in an experiment. When each green maintains its question, its subject and its limit, the group can support a defensible decision.

When everyone just saw “CI passed”, the system loses exactly the precision that engineering was supposed to create.

Previous reading: Engineering harness in practice. Next reading: *Autonomy proportional to risk*, on 08/18 at Trustyu Forge.