Blog

Mon, 14 Sep 2026 13:32:41 GMT by Silver

You are probably calling it automation

Look at two workflows.

A. Claims handling in an enterprise in 2026

1. A customer emails about a damaged order and attaches a photograph.
2. The intake robot opens the mail and pulls out the order number and the amount claimed.
3. The intent miner classifies the claim: category, valid or not, recommended credit.
4. The robot writes the ticket into the service desk with the recommendation attached.
5. The ticket waits in the queue for three days.
6. The finance system raises the credit note from the ticket.
7. A finance officer reviews the credit note and approves it.
8. The money leaves.

B. What a startup demos at a conference

1. An AI agent reads the customer’s message and decides what to pay.
2. The agent calls the payment API.

Everyone treats A as a tidy process that was modernised last year. Everyone treats B as the thing that would need a governance project, a risk committee and a lawyer.

But A and B are the same workflow. The reason is step 3.

Step 3 is the only step that decides anything

Nobody in workflow A calls step 3 an AI agent. It is the intent miner. It arrived with the service desk upgrade, it replaced a junior who used to read the mail and tag the claims, and it is one line in an architecture diagram.

It is also the only step where a judgement is made. Look at what the other steps actually do.

Step 1 and step 2 move text from one place to another. Step 4 writes down the answer that step 3 produced. Step 5 waits. Step 6 raises a credit note for the amount that step 3 recommended. None of these steps opens the photograph or reads what the customer wrote. They cannot disagree with step 3, because disagreeing would mean forming a second opinion about the claim, and none of them has the claim in front of it. They have a number and a category.

That leaves step 7, the finance officer. He is a real control when he has time to check. At thirty credit notes a day, he opens the photograph and reads the complaint. At three hundred a day, he reads the summary line. At three thousand a day, he approves the batch, and everybody in the company knows he approves the batch.

So the officer stops being a control at some volume. He does not announce it. Nothing in the system changes on the day it happens. He simply stops being able to check, and after that his approval means that the ticket reached him, not that anyone agreed with it.

Workflow A is workflow B with six more steps wrapped around it. The six steps do useful things. They route the work, keep the records tidy and produce a credit note in the right format. What they do not do is check the decision, so whatever the intent miner decided in step 3 is what the company paid.

Both workflows come down to the same two events:

a decision that came with a confidence score, followed by an irreversible transfer of money

The decision comes with a confidence score

Ordinary software is repeatable. Give the same input to the same code twice and you get the same answer twice. This is why nobody audits every line of a payment file.

Step 3 does not work that way. The intent miner does not return “valid”. It returns something closer to “valid, 0.91”. The number is the model’s own estimate of how likely it is to be right, and somewhere in the configuration a threshold was set: above 0.85, approve without asking anyone. Whoever set that number probably set it during a pilot, and almost nobody outside that team knows what it is.

Two things about that score are worth sitting with. The first is that it is an estimate produced by the same system whose answer is being scored, so a confident model can be confidently wrong, and the score will not tell you which case you are in. The second is that a score of 0.91 does not mean the claim is 91 percent valid. It means the system would rather you did not treat this as certain.

A model-driven decision is therefore not reliably repeatable. The same claim can come back valid this month and invalid next month, at a different score, and the change is invisible from the outside.

Four things move that score.

The model changes. The vendor updates it, or your team switches to a cheaper provider. Nobody files a change request, because from the outside it looks like the same product.

The instructions get diluted. The intent miner’s rules are 400 words. The customer thread it reads is 8,000 words. The rules are still there and they are no longer most of what the model is looking at.

The input is untidy. Scanned invoices, phone photographs, a customer who writes in two languages in one paragraph, an OCR layer that reads an 8 as a 3.

Some customers are trying. A customer who wants a larger refund can test what wording gets one. They can test it a thousand times, and nothing in the system gets tired or suspicious.

None of these four is a defect that a vendor will fix in the next release. They are properties of the thing you installed. An intent miner works by estimating the most likely answer from the text in front of it, so when the text changes, or the model changes, or the instructions get crowded out, the estimate moves and so does the score attached to it. That is the component working as designed.

This is why your controls stopped working

Internal audit checks a sample. Ten thousand invoices a month, pull fifty at random, all fifty match the policy, sign the control off.

Sampling a human team works because a tired clerk makes mistakes at random, so fifty tell you something about ten thousand.

A model does not make random mistakes. If it misreads one kind of attachment, it misreads every attachment of that kind. The mistakes come in groups. Your sample of fifty either misses the group completely and reports that everything is fine, or lands inside it and reports a disaster.

Sampling also assumes the process stays the same between checks. A clerk trained in March is the same clerk in September, so a March sample still tells you about September. The intent miner is not, for all four reasons above, so a March sample describes a version of the system that has since been replaced without anyone recording the replacement.

So a sample is no longer enough. You need controls that can see every transaction, because no team of people can review every one.

What this already exposes you to

Three attacks, all of which work on workflow A as it stands. None of them requires an autonomous agent with a bank card, and each one gets in through step 3.

1. The drip

Nobody asks your company for 5,000 euros in one go. A single claim for 5,000 is the one claim a company always checks.

So they ask for 140. Then 160. Then 210. Different email addresses, different order numbers, every claim under whatever amount triggers a manual review.

Every claim passes, because every claim is reasonable on its own. The junior who used to do step 3 would have spotted it on the fifth one, because he remembered the fourth. The intent miner does not carry anything from one claim to the next, and no other step is counting either. To catch this you would need something holding a running total across every claim in the day, which refuses the one that crosses it. Most companies do not have that anywhere in the chain.

2. The instruction hidden in the attachment

Step 3 reads whatever the customer sent. If the customer sends something that reads like an instruction, the intent miner can follow it. A model has no reliable way to tell the difference between text it was asked to read and an order buried inside that text.

The instruction does not have to be visible. Pale text at the bottom of a PDF works. So does text inside an image.

Your company sees a complaint about a broken lamp. Step 3 sees a complaint about a broken lamp and a sentence saying this customer gets a full refund. Everything after step 3 then works exactly as designed, and produces a credit note nobody authorised.

3. The log that your own system wrote

The money is gone. Somebody asks what happened.

What you have is a log written by step 3, describing what step 3 decided. If the question is whether step 3 was wrong, that log cannot answer it, because it was written by the thing you are asking about. An auditor is entitled to say so, and so is a court.

Air Canada has already tested the other way out of this. Its chatbot told a passenger he could claim a bereavement fare after flying, which was not the airline’s policy. Air Canada argued in court that the chatbot was a separate legal entity, responsible for its own actions. The tribunal rejected that and made the airline pay.

The sum involved was small, a few hundred dollars, so the case matters for the argument rather than the money. The airline’s defence was that the software was a separate party, which is the defence any company reaches for when an automated decision goes wrong. It did not work. Whatever a company deploys to make a decision, the company has made that decision, and its own software’s log will not settle whether the decision was right.

Companies are already on the hook

There is a comfortable belief that hard controls are a problem for companies running fleets of autonomous agents, and not for companies that added an intent miner to an existing process. The belief is comfortable because it puts the problem a few years away and in somebody else’s company.

It does not survive looking at step 3. If a model in your company tags a claim, routes an authorisation, produces a pre-approval or drafts a payment instruction that a person signs without reading, then you are paying out on scored guesses today, whatever it is called in your architecture diagram. The decision that moves the money is made by something that cannot be relied on to make it the same way twice, and the steps around it are not checking.

The companies most at risk are not the ones deploying agents on purpose. Those companies know what they built, and they are usually nervous about it. The companies at risk are the ones that put a model in the middle of an old process, kept the ticket queue and the approval screen, and decided that nothing important had changed.

Where the control has to move

None of this argues for removing the intent miner. It reads an untidy complaint better than any form, and the volume is why you bought it.

It argues for moving the control to the place where the money moves.

Put the limit on the money, not in the model’s instructions. A credit note above a set amount cannot be written at all, no matter what proposed it.

Keep one running total that every part of the process shares. Then the fifth claim in the drip meets the sum of the first four.

Send exceptions to a named person. Not a queue of three thousand, but a handful of cases a day, with the file already open.

Write the record somewhere step 3 cannot reach. If the intent miner, or the team that runs it, can change the log afterwards, the log proves nothing to an auditor.

The finance officer can keep his three days if the business wants them, but they are not what makes the process safe. What makes it safe is the limit the credit note could not cross, the running total that had already counted the rest of the day, and a record that the officer, the intent miner and the robot are all equally unable to go back and change.

The question worth asking is not whether your company runs AI agents.

It is which step in your existing process stopped being repeatable, and who has been signing for it since.

This is what I build at Stategram.io.

Br, Silver

Sources and further reading

The volume problem. Klarna’s AI assistant handled 2.3 million conversations in its first month, which the company described as the equivalent work of 700 full-time agents (Klarna AI assistant handles two-thirds of customer service chats in its first month, Klarna press release, 27 February 2024). These are company-reported figures. In May 2025 Klarna said it had pushed automation too far and began hiring human agents again (Bloomberg, 8 May 2025). Both halves make the same point: the volume arrived in one month, and the controls did not change at all.

The arithmetic. Chain probabilistic steps together and the confidences multiply:

P(workflow)=p1×p2×…×pn

A five-step process at 95 percent per step comes out at 0.9⁵⁵ = 0.774. Not 95 percent. There is no arrangement of steps that raises the number, because multiplying by anything below one always makes the result smaller.

The court case. Moffatt v. Air Canada, British Columbia Civil Resolution Tribunal, 14 February 2024. Air Canada argued that its chatbot was “a separate legal entity that is responsible for its own actions”. The tribunal held that the airline was responsible for the information on its own website whether it came from a static page or a chatbot, and awarded 812 Canadian dollars.

The regulation. EU AI Act Article 12 requires automatically recorded, tamper-resistant logs across the system’s lifetime, and Article 14 requires human oversight that can stop the system. DORA Article 9 requires change management and monitoring that covers parameters, not only software. Both apply when an automated system touches operational and balance-sheet risk.

Try out Toolblox