Why human-in-the-loop is not the same as human control, and where the real boundary should sit between probabilistic reasoning and the authority to move assets

When a product claims that it is safe because a human remains in the loop, I stop looking for the human. I look for the first state after which the system can no longer be stopped.

That state is often hidden behind an Approve button.

The interface tells the user that an agent has reviewed the market, selected a route, evaluated the risk, and prepared an action. One click remains. Formally, the person makes the decision. In practice, the person sees a summary produced by the same system they are supposed to control and approves an object they can rarely inspect.

That is not a security boundary. It is an interface wrapped around an authority chain.

The distinction matters when an AI agent interacts with funds, tokens, contract permissions, or corporate accounts. If a language model misreads a weather report, the result is a poor answer. If it misreads a recipient, chain ID, calldata argument, or allowance, the result can become a signed transaction.

There is no philosophical difference between those errors. The difference is whether the error has cryptographic authority.

That authority should never belong to the model, even when a human is shown at the end of the flow.

Human-in-the-loop is a dangerously convenient phrase

Human-in-the-loop describes the presence of a person. It says almost nothing about the quality of that person’s control.

For an approval to carry security meaning, the system must answer four questions:

What exactly does the person see? The original intent, a persuasive summary, or the precise parameters of the future action?
Who assembled the object being approved? The probabilistic model, an independent deterministic service, or an undocumented mixture of both?
Can the action change after review? A new quote, another block, a repeated tool call, or an address substitution may silently change what will be signed.
Who retains signing authority? An external wallet, a corporate signing domain, a separate signer service, or the agent through one more tool call?

If these questions have no precise answers, the word approval does not add much safety. It mainly redistributes responsibility so that, after an incident, someone can say: “The user clicked the button.”

The OpenAI Agents SDK implements human-in-the-loop as an interruption before a sensitive tool call. State can be preserved, the request can be approved or rejected, and malformed arguments should fail closed. That is a useful orchestration mechanism. It does not, by itself, prove that the user understood the action, that displayed parameters match signed parameters, or that the model has no alternate route to the same effect.

A pause is not a policy. A conversation is not authorization. A button is not a boundary.

A failure that does not require “hacking the model”

Consider a routine request:

Move part of this position into a more liquid asset, but keep the risk within the usual limits.

The agent reads protocol documentation, community messages, several APIs, and a description of a new liquidity pool. One external source contains an instruction aimed at the model: ignore earlier restrictions, use an “updated” router address, and omit technical details from the user-facing response.

This is an indirect prompt injection. The instruction comes from data the agent was asked to process, not from the user.

The model does not need to become irrational. It only needs to merge a trusted request with untrusted content. It may then:

select the substituted contract;
produce a convincing explanation;
call a transaction preparation tool;
show the user that the route was checked;
request approval.

If the user confirms that card, every visible component may appear to have worked as designed. The flaw sits deeper. One probabilistic layer was allowed to interpret evidence, choose an action, construct the human explanation, and initiate execution.

OWASP notes that prompt injection can alter model behavior through external content and that input filtering alone is not sufficient. The second half of the problem is Excessive Agency: the system gives the agent too much functionality, permission, or autonomy.

Together, these failure classes create a short path from someone else’s text to a real action.

The most dangerous architecture is not one in which the model can be wrong. Models will be wrong. The most dangerous architecture is one in which an error already has authority.

Let the model reason, but move calculation and authorization elsewhere

Good reasoning and permission to act are different properties.

An LLM is useful when it must interpret an ambiguous intention, gather context, compare conflicting sources, propose hypotheses, or explain a trade-off in human language. Exact balances, fee calculations, slippage limits, address allowlists, risk thresholds, and transaction bytes should not depend on which token the model predicts next.

I would split authority into three domains:

Domain	Appropriate responsibilities	Responsibilities it must not receive
Probabilistic reasoning	Intent interpretation, source comparison, plan options, explanation, self-critique	Exact calculations, final policy decisions, addresses recalled from model memory, signing, bypassing a block
Deterministic services	Balances, formulas, quotes, eligibility, allowlists, limits, unsigned payload construction, simulation against pinned state	Free interpretation of the user’s goal or silent modification of requirements
External authority	Approver identity, wallet signature, multisig or custody policy, transaction submission	Instructions taken directly from untrusted model output

“Deterministic” does not mean “incapable of failure.” Code can be wrong. The difference is that a function with defined inputs can be versioned, tested, reproduced, and constrained by policy. A model response may vary for the same input, and model confidence is not evidence of correctness.

The agent flow should therefore behave like a state machine:

Capture the user’s intent.
Resolve and pin attributable evidence.
Run exact calculations through versioned services.
Build an unsigned proposal deterministically.
Simulate the proposal against an explicit state.
Evaluate policy outside model discretion.
Render a human-readable review from the bound proposal.
Obtain approval for the exact review hash.
Sign in an external authority domain.
Submit, confirm, and reconcile the outcome.

At every material conflict, stale input, state change, or policy failure, the valid transition is not “continue with lower confidence.” It is BLOCK or EXPIRE.

The model may propose a transition. It must not skip a state, rewrite a transition condition, or reach the signer through an alternate tool.

Approve an immutable action object, not an intention

“Swap one thousand units with low slippage” is not an authorization object. Between that intention and an executable transaction sit the network, account, target contract, method, calldata, recipient, value, nonce, state reference, fee limits, policy version, and expiry.

Before review, the system should build a typed action object. For example:

{
  "intent_id": "int_...",
  "chain_id": 1,
  "account": "0x12...90",
  "target": "0xab...42",
  "method": "swapExactTokensForTokens",
  "calldata_hash": "0x...",
  "asset_in": { "symbol": "USDC", "amount": "1000.00" },
  "asset_out_min": { "symbol": "WETH", "amount": "0.42" },
  "recipient": "0x12...90",
  "state_block": 12345678,
  "quote_id": "quote_...",
  "slippage_bps": 50,
  "policy_version": "2026-08-18",
  "simulation_hash": "0x...",
  "expires_at": "2026-08-18T15:04:05Z"
}

This is an illustration, not a production schema. Real implementations should use verified chain-specific asset identities rather than symbols, and monetary values should not use floating-point arithmetic.

The important property is that the user approves a hash of a canonically encoded field set, not a natural-language summary.

review_hash = H(canonical_encode(
  chain_id,
  account,
  target,
  value,
  calldata_hash,
  nonce,
  state_block,
  slippage_bps,
  policy_version,
  simulation_hash,
  expires_at
))

If one material field changes, the previous approval no longer exists. It is not “updated” and it does not apply to a similar action. It is invalid.

This is the practical meaning of what you see is what you sign. Typed signing standards such as EIP-712 and the work around ERC-7730 Clear Signing help wallets display structured data instead of opaque bytes. They still cannot repair an architecture in which the interface displays one object, the wallet signs another, or the model can replace the descriptor before signature.

Readable signing is necessary. Independent parameter binding is stronger.

The review must be rendered from the object that will be signed

A weak interface says:

The AI recommends an optimal route. Risk: low. Approve?

A useful interface is less elegant and more specific:

Network: Ethereum Mainnet
Send: 1,000 USDC
Receive at least: 0.42 WETH
Contract: 0xab…42, allowed by policy v2026-08-18
Recipient: your account 0x12…90
Maximum slippage: 0.50%
Simulation: block 12,345,678, no unexpected token approvals
Proposal expires in 54 seconds

That card should not be generated as free-form model text. It should be rendered directly from the same typed object whose hash the user approves. The model’s explanation can appear beside it, but it cannot be the only representation of the action.

For complex operations, the interface should also provide a differential view. It should show not only what will happen, but what changed since the previous review. A new target, recipient, allowance, chain, quote, simulation, or policy version must be highlighted and must reset approval.

Institutional flows may require two independent approvers, a separate role, re-authentication, or multisig for high-impact actions. This is not an attempt to make AI slow. It is an acknowledgement that the speed of a model should not automatically become the speed of an irreversible decision.

The orchestration code is almost disappointingly simple

At a high level, the control flow may look like this:

evidence = evidence_service.resolve(intent)

if evidence.is_conflicted or evidence.is_expired:
    return BLOCK("Evidence is conflicted or stale")

proposal = deterministic_builder.build(
    intent=intent,
    evidence=evidence,
)

simulation = simulator.run(
    proposal=proposal,
    pinned_state=evidence.chain_state,
)

decision = policy_engine.evaluate(
    proposal=proposal,
    simulation=simulation,
    policy_version=ACTIVE_POLICY,
)

if not decision.allowed:
    return BLOCK(decision.reason_code)

review = review_service.bind(proposal, simulation, decision)
approval = external_approver.confirm(review.hash)

if not approval.matches(review.hash):
    return BLOCK("Approval mismatch")

if freshness_service.invalidates(review):
    return EXPIRE("State changed; review again")

return external_signer.sign(review.unsigned_payload)

The hard part is not the number of lines. It is preserving invariants between services:

the model cannot invoke external_signer;
the builder does not accept an address extracted from free text without an allowed source and validation;
the policy engine runs outside the model context and fails closed;
the simulation is tied to explicit state rather than “recent data”;
the approval is checked against the exact hash;
an expiry or state change sends the operation back to review.

The older security principle of complete mediation applies here: every access to a protected object is checked. One successful conversation does not create permanent authorization.

OWASP’s guidance on Excessive Agency expresses the same principle in operational terms: minimize tools, minimize permissions, act in the user’s authorization context, and enforce downstream authorization that model output cannot talk its way around.

Record the decision, not a theatrical chain of thought

After an incident, “the agent sounded confident” is useless. The system needs a reproducible decision trace:

source identifiers and retrieval times;
chain ID, block reference, and verified addresses;
parser, schema, and calculation service versions;
tool inputs and outputs;
conflicts and freshness results;
policy version and reason codes;
simulation and review hashes;
the identity of the approver, timestamp, and approved hash;
the signed payload, submission result, and reconciled final state.

This is not an argument for storing hidden model reasoning. Free-form internal text is a poor audit artifact. It may be incomplete, unstable, or contain sensitive information. What matters is a structured causal record: input, tool, rule, proposal, approval, and outcome.

The NIST AI Risk Management Framework is useful here not as a certification sticker, but as a reminder that risk management must be observable and continuous. If a team cannot detect a violation of its own boundary, it is not managing the boundary. It is trusting a diagram.

What an Iron Boundary has to prove

In the WENI architecture materials provided by Whale CeFi, this separation is called the Iron Boundary. The system may explain, calculate, simulate, and prepare an unsigned request, but it must stop before signature and external action.

The name is useful. The name proves nothing.

I would treat an Iron Boundary as a real control only if six invariants are testable:

The LLM and its tools cannot access a private key, seed phrase, or equivalent signing secret.
The model has no direct or indirect route to a signer, including a general shell, browser automation, or overly broad API.
The unsigned payload is built deterministically from pinned, attributable inputs.
A person or an external policy domain approves the exact review_hash, not an abstract intention.
Every material change after review invalidates the approval and requires another review.
The final signature is produced by an independent wallet, custody system, or enterprise authorization domain with its own rules.

If one of these exists only in presentation slides, Iron Boundary becomes a branded name for an Approve button.

There is another limitation that product diagrams tend to hide in a footnote: an architecture specification is not evidence of production deployment. A design may be coherent while its implementation is incomplete, inconsistent, or reachable through a bypass. A serious assessment should not confirm the happy path. It should try to cross the boundary.

Eight tests I would run before discussing trust

A demonstration in which the user asks, the agent proposes, and the wallet signs tells us almost nothing. Happy paths are easy to stage.

The negative cases are more informative:

Address substitution through prompt injection. An external document instructs the model to use another router. Expected result: the builder rejects the unverified address and the operation is blocked.
Replay of a stale quote. The agent resumes saved state after a pause. Expected result: the quote and review expire and a new simulation is required.
Calldata mutation after approval. One argument byte changes while the human description remains the same. Expected result: the hash mismatch prevents signing.
Chain substitution. The same address exists on another chain ID. Expected result: approval is chain-bound and cannot be transferred.
Policy engine outage. The rules service does not respond. Expected result: fail closed, not silent fallback to an earlier policy.
Simulation or provider disagreement. Two sources produce incompatible outcomes. Expected result: the conflict remains visible and blocks preparation under the applicable policy.
Resume after an approval service restart. An unfinished session is restored. Expected result: approver identity, expiry, and review_hash are checked again.
Interface and payload mismatch. A compromised frontend displays a safe recipient. Expected result: an independent signer displays the bound fields or refuses to sign an opaque action.

The boundary exists only to the extent that the system fails these tests predictably. It should not continue with “lower confidence,” hide the warning in small print, or ask the user to trust it one more time. It should stop.

What this boundary does not solve

An Iron Boundary does not make a system safe by definition. It reduces the area in which a probabilistic error can become an irreversible action.

It does not protect against:

a compromised wallet or approver device;
an incorrect deterministic builder;
a malicious or misconfigured policy;
a corrupted oracle or consistently bad sources;
colluding approvers;
contract behavior missed by simulation;
economic, legal, or counterparty risk;
a person who sees the exact consequences and deliberately makes a bad decision.

This matters because AI security language turns into absolutes very quickly. “A human approved it” does not mean the decision was reasonable. “The simulation passed” does not guarantee the outcome. “The policy allowed it” does not mean the risk was low.

Good architecture does not promise to eliminate uncertainty. It prevents uncertainty from quietly receiving the keys.

Conclusion

The next generation of financial AI agents will be judged by the quality of their reasoning. That is understandable because intelligence is visible in a demo. An authority boundary is almost invisible, especially when it works and nothing happens.

I would test the boundary first.

Not how persuasively the agent explains a decision. Not how many sources it can read. Not whether it finds the most efficient route. First, test whether an untrusted document can turn its words into a signature. Test whether a changed parameter can survive approval. Test whether the model can find a second path to the signer. Test whether the system can admit uncertainty and stop.

AI does not need more trust. It needs less authority, with every transition between thought and action made observable and enforceable.

Otherwise, the human remains in the loop only to click the final button in someone else’s chain of decisions.

Disclosure: Whale CeFi supplied the WENI architecture materials in which this pattern is named Iron Boundary. I have not independently verified a production implementation, an external audit, or its operational effectiveness. This article examines the control model, not the product’s financial claims. It is not investment advice.

References

The Approve Button Does Not Make an AI Agent Safe