A security prompt is not a control: how to prove properties in AI-generated software

Writing “use best practices” or “generate secure code” in the prompt can guide the model. By itself, it does not create a verifiable property in the delivered system. The same generator that received the instruction cannot be the sole authority declaring that it complied.

This Evidence Brief part of the paper Prompt Structure Redistributes, Not Reduces to separate four layers that are often mixed together: generation guidance, produced behavior, tool finding, and accepted risk. The study supports the need for this separation; the technical contract below is a professional proposal from Trustyu Forge, not an experimental conclusion from the authors.

What was measured

The researchers used 424 Python tasks from a synthetic vulnerability-oriented dataset. Each task ran once through five prompt variants and two models — GPT-4o and LLaMA 3.1-8B — in a zero-shot design. That totals 4,240 interactions. Outputs containing valid Python were analyzed with Bandit and CodeQL.

The first variable was generation compliance, not security. With GPT-4o, invalid outputs fell from 338 across 424 tasks under the minimal prompt to 37–52 under the structured prompts. The response was less stable with LLaMA. A structured prompt therefore substantially changed the likelihood of having code to analyze — and changed the denominator for the comparisons that followed.

Conditioned on valid code, security guidance did not consistently reduce the overall prevalence of detected weaknesses. With GPT-4o, high-severity findings moved from 20.8% to 13.6%, while low-severity findings rose from 32% to 43.5%. LLaMA's effects were less consistent. The observed result was model-dependent redistribution, not uniform elimination.

What these numbers do not prove

• the corpus does not represent real production applications;
• Python, two models, and one generation per configuration do not support universal generalization;
• Bandit and CodeQL findings are static signals, not confirmation of exploitability;
• static tools can produce false positives and false negatives;
• the study did not run a broad program of functional and dynamic tests;
• the manual audit of relevance and drift covered 15 of the 424 tasks;
• the replication package is available, but there was no independent replication as of this brief's cutoff.

The paper itself reports relevant behaviors in outputs that neither tool flagged and states that runtime, context, and logic require complementary analysis. A green result reduces one class of uncertainty; it does not conclude the security analysis.

Compliance, correctness, and security are different gates

A prompt can influence the first gate and help with the following ones. It must not have the authority to waive them. [HARNESS31-C2]

Semantic drift must also be tested

In the paper's manual sample, stronger security prompts led GPT-4o to replace or remove explicitly requested insecure constructs. Semantic drift rose from 3/15 in the structured baseline to 9–10/15 in later variants. Because the benchmark contains deliberately dangerous requests, changing the implementation may be desirable. Even so, reducing findings by changing behavior does not prove that the implementation preserved the product decision.

Security and fidelity to the spec therefore should not compete inside the prompt text. The spec states the authorized behavior and its invariants. The threat model states abuse, assets, and boundaries. Tests demonstrate both through separate mechanisms.

Technical contract: guidance cannot become authority

1. Specification before generation

Record functionality, data, identities, capabilities, dependencies, limits, prohibited cases, and acceptance criteria. Ambiguous prompt text must remain an explicit risk, not a requirement implicitly “understood” by the model.

2. Controls outside the model

Authorization, isolation, secrets, network policies, resource limits, data protection, and release gates must exist as platform mechanisms. The model may suggest configuration; it must not silently rewrite the policy that evaluates its own output.

A separate workspace helps reduce collisions, but it is not equivalent to a sandbox with controlled egress, identity, capabilities, and resources. [R1-C2]

3. Multi-method verification

Combine mechanisms according to risk:

• lint, types, and build for basic integrity;
• unit, contract, and regression tests for behavior;
• SAST and semantic analysis for known patterns and flows;
• SCA, SBOM, and policy for dependencies and provenance;
• negative tests, fuzzing, or DAST when warranted by the surface;
• specialized review for identity, authorization, cryptography, data, and infrastructure;
• evals when model behavior remains inside the product.

No individual tool is a certificate. Divergences, suppressions, and uncovered areas belong in the risk record.

4. Authority and segregation

The generator may trigger tests and propose fixes. The verifier needs protected criteria. The person accepting risk must be identified and authorized. Distinct roles create a counterpoint only when permissions, inputs, findings, and decisions are preserved; different names in a prompt do not prove independence. [R1-C3]

5. Release tied to the exact artifact

Checks must point to the evaluated commit. The build must produce an artifact identified by digest. Approval must reference that same target, and promotion must produce a receipt. If a new push changes the code, previous evidence no longer authorizes the release.

Minimum Evidence Packet

For every AI-assisted change with material impact, preserve:

• spec, threat model, and the decision that initiated the work;
• provider/model when available, configuration, and hash of the reusable prompt;
• base commit, diff, and executor identity;
• verifier versions and configurations;
• executed tests, results, and relevant raw artifacts;
• findings confirmed, dismissed, mitigated, or accepted, with justification;
• suppressions and exceptions with an owner and validity period;
• human approval tied to the commit and artifact digest;
• promotion receipt, smoke test, observability, and rollback;
• known limitations and next revalidation.

If the organization can show only the “secure” prompt, it has an intention. If it can show controls, results, authority, and the artifact, it has evidence.

Risk-proportional gate

The table is professional guidance, not a result from the paper. The exact profile depends on impact, exposure, reversibility, and product obligations.

What this brief proves — and what does not prove

The study provides evidence that prompt structure can change compliance, severity, and the composition of findings in a controlled benchmark. It does not prove that prompts are useless, that any model is universally insecure, or that the identified findings would be exploitable in production.

The brief proposes keeping security prompts in the flow as defense-in-depth guidance. A security claim, however, must rely on specified properties, protected verifiers, explicit authority, and evidence tied to the exact software.

Direct sources

• Paper on arXiv
• Full HTML version
• Replication package on Zenodo
• Source dataset
• Bandit
• CodeQL
• NIST SP 800-218 — Secure Software Development Framework

Conclusion

“Make it secure” is a useful instruction. “It was verified, approved, and promoted under these controls” is a governable statement.

In AI-native engineering, the prompt starts the work. Architecture, tests, authority, and the Evidence Packet determine whether the result may operate.