Autonomy proportional to risk: where to delegate to AI and where to keep the human gate

“How much autonomy should we give AI?” It looks like a question about trust in the model. In practice, it is a question about the system: what action can occur, on what data, in what environment, with what effect, reversibility, supervision and evidence.

The same model can be given broad autonomy to organize personal notes and no authority to approve credit, publish a commercial promise or remove data. Capacity does not determine grant. Declared trust does not replace control.

This article presents a practical way to scale autonomy without falling to the extremes of blocking all use or handing over irreversible decisions to a probabilistic executor.

Executive summary

The NIST AI RMF organizes risk management as an ongoing activity throughout the lifecycle, with governance spanning mapping, measuring and managing. He also recommends differentiating roles and responsibilities in human–AI settings. THE OWASP AISVS, on the other hand, offers testable security requirements for AI-enabled systems. Neither provides universal authorization for a specific agent; the exact profile remains environment and threat model dependent. [R2-C1] [R2-C3]

The FORGE proposal is to link autonomy to four axes: effect, reversibility, sensitivity and uncertainty. The larger any axis, the more explicit policy, verification, evidence and human gate need to be. The human doesn't need to click on everything; needs to retain authority where error is difficult to detect, attribute or reverse.

Autonomy is not intelligence

Intelligence describes the ability to produce or select actions. Autonomy describes authority to execute them without new approval. Mixing the two creates the dangerous argument: “the model has improved, so it can do more”.

Responsible promotion asks:

• Does the system know the task limit?
• Is the executing identity verifiable?
• tools and data use allowlist?
• Is there a budget for action, time and cost?
• Is the effect observable and reversible?
• Are critical failures detected before leaving the border?
• Who accepts the residual risk?

The model participates in the execution. The organization continues to own the authority.

The four axes of action risk

Effect

A private suggestion and a message sent to the customer do not have the same effect. Read, write, publish, transact, and destroy as different classes, even when they belong to the same stream.

Reversibility

Reverting a draft is simple. Recovering an exposed secret, a financial transfer or a regulatory decision may be impossible. The smaller the reversal window, the sooner the gate should appear.

Sensitivity

Public, internal, personal, financial, legal and credential data require distinct boundaries. Access to data does not imply permission to transform or send it.

Uncertainty

Tasks with objective criteria and a stable environment support more automation. Ambiguity, exceptions, context change, and subjective evaluation increase the need for human containment and sampling.

A five-level operating scale

Level 4 does not mean “forbidden forever”. It means that unattended execution is not the default. To reduce gate, the system needs to demonstrate controls, history, and the ability to detect failures in the exact scope — not just present a newer model.

The autonomy contract

Before handing over a tool to the agent, record:

The contract must be business-readable and technology-enforceable. “Use common sense” is not policy.

Human gates that really help

A poorly designed human gate transfers noise: dozens of approvals, insufficient context, and pressure to click. The goal is not to maximize clicks; is to place judgment at the point where he changes the decision.

A good approval request shows:

• proposed action and recipient;
• motive and evidence used;
• data that will leave the border;
• alternatives considered;
• expected effects and rollback;
• checks already carried out;
• residual risk that the person is accepting.

If the person needs to reconstruct the entire work, the agent has not prepared a decision; just forwarded a task.

Shadow before authority

For risk-relevant flows, execute first with no external effect. Compare the proposed action with the real process, record divergences and test stop conditions. Shadow produces evidence without implicitly promoting authority.

A mature sequence might be:

1. replay in synthetic fixture;
2. shadow with data allowed and no effect;
3. reversible execution with narrow scope;
4. canary with allowlist and supervision;
5. expansion conditioned on metrics and incidents;
6. Automatic rollback when the boundary fails.

Elapsed time does not promote the system. Reviewed evidence can promote.

The delegation matrix

The last example shows why “high effect = always human” is also simplistic. In incident response, a critical, delimited and pre-authorized action may need to be automatic. The contract is more precise than intuition.

Metrics that justify expanding authority

Don't just use average success rate. Please note:

• critical failure by action class;
• detection before versus after the effect;
• rollback time and rate;
• divergence between proposal and human decision;
• exceptions outside the contract;
• concentration of failures by population or data;
• cost of supervision;
• near-incidents and triggered stop conditions.

A correct stop is a sign of control, not necessarily a product failure.

What this article proves — and what it doesn't prove

NIST and OWASP support ongoing risk management, explicit responsibilities, and verifiable requirements. The FORGE canonical sources also outline which controls need to be adjusted to the environment. [R2-C1] [R2-C3]

The five-level scale and four axes are an operational synthesis of FORGE. They do not replace legal, regulatory, security, or domain impact assessment. They also do not guarantee absence of damage. They serve to prevent an authority decision from being made solely based on the model name or a successful demonstration.

Conclusion

Safe autonomy is not the absence of humans. It is explicit distribution of authority.

AI can perform more and more, but the organization needs to decide where the effect starts, who accepts the risk, how the action is verified and what evidence allows it to extend or revoke the grant. The goal is not to keep people on every click; is to maintain responsibility for irreversible points.

Previous reading: Spec-driven without self-deception. Next reading: *Evidence before adjective*, on 08/20 at Trustyu Forge.