Evidence before adjective: how to defend claims about AI systems

“Secure”, “reliable”, “auditable”, “autonomous” and “world-class” seem to sum up maturity. Without a verifiable question, however, these adjectives merely shift the work to the reader: secure against what threat, reliable in what task, auditable by whom, autonomous to do what?

A responsible claim does not attempt to eliminate all uncertainty. It informs exactly what was observed, about what object, by what mechanism, under what limits and until when the evidence remains valid.

This article proposes a practical grammar to transform language confidence into inspectable confidence.

Executive summary

Provenance is verifiable information about where, when and how an artifact was produced. A specification SLSA organizes attestation levels and formats to increase supply chain guarantees. This strengthens the origin and integrity of the declared process, but does not prove the semantic correctness of the artifact. [R2-C2]

Likewise, a scanner, test, eval, review, or audit examines specific properties. The defendable claim needs to call subject → property → mechanism → evidence → verifier → validity → limit of inference. If any link is missing, the adjective must decrease in strength.

The goal is not to write more disclaimers. It’s about allowing the client, engineer, auditor or board to understand what they can decide without reconstructing the whole story.

The anatomy of a claim

Consider: “The release is secure.” The phrase does not identify object, threat, control or verifier. A more defensible version would be:

The image sha256:… was produced by workflow release.yml in the approved builder, has signed provenance linked to commit X and passed scanner Y without HIGH findings in policy Z on 08/20. This does not prove the absence of vulnerabilities or security of runtime.

The claim got bigger, but it also became usable. It contains:

Provenance, attestation and verification

Provenance describes the relevant production history. Attestation is an authenticated statement about a subject. Verification compares this statement and its artifact to expectations and roots of trust.

The three things are not synonymous:

• unsigned provenance may help diagnosis but offers limited warranty;
• attestation signed by an untrusted identity preserves a statement you do not accept;
• valid signature with wrong subject authenticates the wrong link;
• correct provenance does not state that the business rule is correct;
• checking without explicit policy can only confirm structure.

SLSA recommends binding provenance to the artifact and checking subject, signature, builder and parameters against expectations. Still, the very canonical boundary of FORGE remains: attestation proves the declared string, not semantic correctness. [R2-C2]

Five classes of evidence

Evidence of intent

Spec, ADR, issue and acceptance criteria show what should happen. They are essential, but they do not demonstrate execution.

Evidence of execution

Logs, traces, diffs and receipts show observed actions. Without identity and binding, they can belong to another subject or context.

Evidence of ownership

Tests, scanners and evals check a property under a method. Its power ends in the coverage and validity of that method.

Evidence of provenance

Digests, signatures, attestations, and history link artifact, origin, and process. They do not replace evidence of behavior.

Evidence of outcome

Customer acceptance, perceived quality, incidents and flow metrics show effect. Outcome without causality or baseline still requires caution when attributing change to AI.

A strong claim combines the necessary classes, not all of them per ritual.

The verifier is part of the claim

“Reviewed” does not say who reviewed it, against what criteria or with what independence. The same team that produces the artifact can do valuable self-review; the mistake is to present it as independent assurance.

Declare the mode:

• deterministic automatic verification;
• evaluation by model;
• human author review;
• review by a specialist other than the author;
• independent audit with scope and contract;
• attestation of the platform that carried out the process.

Independence is not a feeling. It is a property of identity, authority, incentive, and access to evidence.

Validity: evidence also ages

A report belongs to a version, an environment, a policy and a date. After changing the model, prompt, tool, dependency, data, or authority, some of the evidence may no longer represent the system.

Each claim must define expiration events:

• new subject or commit;
• change of material configuration;
• external source updated or removed;
• temporal maturity;
• incident that contradicts the conclusion;
• discovery of failure in the benchmark or verifier.

Correction must not delete the previous claim. Must register new version, reason and scope.

The minimum Evidence Pack

For a release decision, a lean package might contain:

1. subject and digest;
2. spec and accepted criteria;
3. applicable policies;
4. results of gates linked to the same subject;
5. provenance and identity of the builder;
6. exceptions with owner and expiration;
7. nominal acceptance decision;
8. public claims with inference boundary;
9. path of correction or withdrawal.

The package is not a huge file. It is a verifiable index to evidence that already exists.

Writing public claims

Swap:

• “completely safe” as “passed controls X and Y in scope Z; risks A and B remain”;
• “validated” by “evaluated in bank V, version N, with result R”;
• “auditable” by “actions A–D preserve subject, actor, timestamp and receipt for 90 days”;
• “autonomous” as “can perform O1/O2 operations in the allowlist, with budget and revocation”;
• “world-class” by comparable reference or remove the adjective.

Accuracy does not reduce commercial strength. It allows confidence to survive the first technical question.

A five-minute test

When reviewing a page, pitch, or report, highlight each trustworthy adjective and ask:

1. What is the subject?
2. What observable property supports the word?
3. Where is the evidence?
4. Who checked and with what independence?
5. What does the sentence not allow to conclude?

If the answers do not fit into a few lines, the claim is still above evidence.

What this article proves — and what it doesn't prove

The SLSA specification and the canonical bundle support the function of provenance and its separation from semantic correctness. [R2-C2] The claims grammar and the Evidence Pack are operational summaries of FORGE supported by its controls, conformance and assurance contracts.

No Evidence Pack proves the absolute absence of risk. An attestation may be valid and the policy insufficient. An independent audit may be narrow in scope. The goal is to make visible boundaries and assignable decisions, not to produce total certainty.

Conclusion

Trust begins when the adjective ends and the evidence can be inspected.

A defensible claim fixes the subject, names the property, shows the mechanism, preserves the evidence, identifies the verifier, and limits validity and inference. This contract allows us to say “we don’t know yet” without paralyzing the work — and to say “it’s ready” without relying solely on reputation.

Previous reading: Autonomy proportional to risk. Next reading: *The agent cannot be its own auditor*, on 08/22 at Trustyu Forge.