AI Agents: towards controlled autonomy

An AI agent differs from an assistant in one simple way: it does not wait for an instruction at every step. You give it a goal, it decides for itself the sequence of actions to take, calls tools, evaluates results and continues until it judges the job done. That is a considerable shift — and the important word in that description is not "autonomous", but "judges".

Current enthusiasm for agents rests on an appealing promise: automating not tasks but entire processes. What we observe in the field is more mixed, and worth stating plainly, because it shapes the investment decisions of the next eighteen months.

The first observation is that an agent's reliability decays with the number of steps. A system that performs one action correctly ninety-five per cent of the time looks excellent. Chained across ten dependent steps, it produces a correct outcome roughly sixty per cent of the time. Across twenty, a third. That arithmetic is not a design flaw the next generation of models will fix: it is the mechanical consequence of composition. It leads to a practical conclusion — effective agents are those whose depth has been deliberately capped.

The second observation concerns failure mode. Conventional software that fails stops and says so. An agent that fails keeps going. It interprets an unexpected result, adapts, continues in a direction nobody anticipated, and produces at the end a plausible report on work that was not done properly. That is a difference in kind, not degree: silent failure is far more dangerous than a loud error, because it raises no alarm.

The third observation is about permissions. A useful agent acts — it writes to systems, sends messages, commits resources. Every capability granted widens the surface of what it can get wrong. The relevant security question is not "is the model reliable?" but "what is the worst possible outcome if this agent is completely wrong, and is that outcome acceptable?".

Here is a typical sequence. It describes no identifiable organisation, but a chain of events observed in several forms. A finance department deploys an agent to reconcile supplier invoices against purchase orders, authorised to clear discrepancies below a threshold. It works remarkably well for several weeks. Then a supplier changes its invoice format. The agent, no longer finding the expected field, falls back on another numerically plausible field and carries on clearing. The anomaly only surfaces at the monthly bank reconciliation. The problem is not that the agent got something wrong: it is that it kept working confidently after it stopped understanding what it was reading.

That case illustrates the principle we apply consistently: an agent must be designed to stop, not only to succeed. At KAIROS Impulse we do not approach these projects by asking "what can the agent do?" but the reverse: under what circumstances must it refuse to act and hand back control? That means defining explicit exit conditions, confidence thresholds, and a human checkpoint placed where the decision becomes irreversible — not everywhere, which would cancel the benefit, but precisely there.

Two families of agents also need separating, and public debate conflates them. Closed-scope agents operate over a defined set of tools, with predictable steps and a bounded objective: processing a leave request, preparing a renewal file, extracting and checking information across a known document set. They are mature and profitable today. Open-scope agents, meant to cope with a general objective using generic tools, remain largely demonstration material. The distance between the two is considerable, and part of today's disappointment comes from organisations that invested in the second category believing they were buying the first.

A fair objection arises: by constraining an agent this heavily, are we not reducing it to a conventional automation, without the benefit of intelligence? The answer lies in what the agent actually contributes. Its value is not deciding freely, but handling variability at the input — understanding an email phrased ten different ways, processing a document whose structure varies, recognising that a case is out of the ordinary. That is precisely what conventional automation cannot do. Constraining actions while leaving interpretive flexibility at the input is not a compromise: it is the correct design.

Three questions before committing to an agent project. How many sequential steps does this process involve, and have we calculated the compounded reliability that follows? What is the worst possible consequence of a wrong action, and is it reversible? And finally, how will we know the agent got it wrong — what signal will reach us, and how quickly? If the third answer is "we'll notice", the project is not ready.