Executive reading · ~60 seconds

Retry recovers a try; Learning changes the probability of recurrence. A material failure closes when it produces mechanism, regression, owner, validity and evidence of effectiveness.

An agent fails, receives a new prompt, and gets it right on the second try. The incident is marked as resolved. Days later, another session encounters the same situation, repeats the error and again consumes time, tokens and human review.

The system corrected an execution. He didn't learn.

Executive summary

Retry restores an attempt. Learning changes the probability of recurrence. A loop closes when failure produces at least one durable mechanism:

  • case of regression;
  • eval or versioned rubric;
  • new or reinforced control;
  • change of context, tool or authority;
  • addressable operational documentation;
  • hypothesis discarded;
  • monitoring capable of detecting repetition.

Not every failure deserves a major retrospective. Every material failure deserves an explicit decision about recurrence, impact and destination of learning.

Five answers that feel like learning

Try again

A retry is correct for transient failure. It becomes noise when it hides deterministic error, missing authority, or invalid input.

Increase the prompt

Adding instructions may resolve the current case and degrade others. Without eval, we don't know if the change generalizes or just memorized the example.

Change the model

A more capable model can reduce a certain class of error. It also changes cost, latency, behavior and safeguards. Upgrade does not replace failed property analysis.

Record a note

Documentation without recovery path, owner and trigger depends on accidental discovery. Learning exists, but it is not connected to execution.

Blame the agent

“The model hallucinated” describes a broad symptom. It does not answer why the system accepted the output, what check was missed or what authority allowed the effect.

From failure to mechanism

Use a simple string:

evento -> impacto -> classe -> causa controlável -> mecanismo -> regressão -> owner -> validade

Event

Register the subject: session, input, model, tool, version, environment and output. Do not copy sensitive data that is unnecessary for learning.

Impact

Separate reversible internal failure from external effect, altered data, information exposure, financial decision or damage to the user. The depth of the response follows the risk.

Class

A small taxonomy helps you find recurrence:

ClassExampleCandidate Durable Response
intentionambiguous taskspec and acceptance criteria
contextmissing or old informationversioned source and validity
policyaction outside authoritydeny-by-default and gate
executiontool failedlimited retry and fallback
verificationchecker did not cover the classnegative test/eval
evidenceresult without bindingreceipt linked to subject
feedbackcorrection did not propagatelearning item and owner

Controllable cause

It is not necessary to discover a metaphysical cause of the model. Look for the boundary that the system can change: input, tool, contract, isolation, verification, authority or operation.

Mechanism

Transform the lesson into an executable or operational artifact. A recommendation without owner and trigger remains a good intention.

Regression

Reproduce the event in a sanitized way. A deterministic test is preferable when it exists; eval with initialing comes in when the property requires judgment. The quality claim needs explicit criteria and observable evidence; trace or subjective inspection alone are incomplete. [R3-C1]

The learning item as a unit of change

A useful log contains:

  • stable identifier;
  • sanitized event and evidence;
  • class and risk;
  • decision made;
  • mechanism or justification for not changing;
  • affected repositories, products or controls;
  • owner and term;
  • regression testing;
  • status: candidate, accepted, implemented, verified or retired.

This unity prevents two extremes: forgetting the failure or transforming any detail into a global rule.

When not to generalize

Learning can also be about keeping the system as it is. An isolated, low-impact and highly unlikely case may not justify permanent complexity.

Record the no-change with:

  • estimated impact and frequency;
  • cost of control;
  • signal that reopens the decision;
  • validity of the conclusion.

“No change” is a decision. “We didn’t do anything” is an absence of process.

The danger of the learning backlog

Extensive catalogs can become an archive. Three guardrails keep the loop alive:

  1. applicability: each item points to the execution point that must change;
  2. priority by risk and recurrence: not on a first-come, first-served basis;
  3. verification: implemented is not synonymous with effective.

Useful metrics include recurrence by class, time to engine, expired items, and proportion of material failures regressing. Raw number of lessons encourages documentation, not learning.

A fifteen-minute ritual

For a material failure, respond:

  1. what happened and which subject was active?
  2. What effect occurred or almost occurred?
  3. where could the system have detected it earlier?
  4. What changes in a durable way?
  5. How will we prove that the change works?
  6. Who responds and when will we revisit?

If there is no answer to 4 and 5, the stream probably fixed the episode, not the system.

What this article proves — and what it doesn't prove

Incident response, evals and continuous improvement practices support the separation between restoring and learning. FORGE materializes this separation in records, gates and regressions. The article does not prove that every error can be eliminated, nor that more rules always increase reliability.

Sociotechnical systems change; models, people and processes share flaws. The goal is to reduce preventable recurrence and make residual risk visible.

Conclusion

Trying again may be the correct operational response. It just shouldn't be confused with learning.

The system learns when an experience changes its future behavior in a way that is addressable, testable, and proportionate to the risk. Without this, each new session pays again for the same lesson.

Previous reading: From checklist to executable control. Next suggested reading: The agent cannot be its own auditor.

Editorial and responsibility note

Research cutoff
Last review
Recorded corrections
No corrections recorded.

The cutoff above applies to the canonical claims. Additional sources and their access dates are identified in the article body.

This article combines cited sources, analysis, and the author's professional experience. Verifiable data and factual statements are linked to their respective sources. Interpretations, hypotheses, projections, recommendations, and opinions represent the author's professional point of view at the time of publication; they do not constitute proven facts, a promise of results, or legal, financial, or technical advice applicable to a specific case. Consult the original sources and qualified professionals before making decisions.

Claims and sources

R3-C1

An AI quality claim requires explicit eval criteria and observable execution evidence; traces alone and subjective inspection alone are incomplete.

Limit: Neither source defines FORGE-specific pass thresholds; baselines must be calibrated per use case. This is a research-synthesis design input; it does not prove product adoption, operational maturity, independent attestation or outcome.

  • Anthropic — Anthropic, MIT repository; analysis-only excerpts
  • OpenAI — OpenAI, MIT repository; analysis-only excerpts