The most expensive bug is overconfidence

The system you think you have

Every organisation runs two systems. The one that exists, and the one people believe exists.

Most of the time the gap between them is small enough not to matter. The belief is close enough to the reality that decisions made against the belief still work out. That is exactly what makes it hard to notice — an inaccurate model of your own operation produces correct decisions for years before it produces an expensive one.

Then something moves. A person leaves. A supplier changes a file format. Volume doubles. A dependency updates itself over a weekend. And the gap that never mattered becomes the entire problem, not because the system failed but because every decision made about it was made against a version of it that stopped being true some time ago.

We walk into operations for a living, and this is the pattern underneath nearly everything expensive we find. Not bad technology. Not careless people. A confident, senior, experienced account of how a system works that turns out to be an accurate description of how it worked three years ago.

Confidence is measured differently to competence

There is a well-known finding usually reached for at this point, and it is worth handling carefully.

In 1999, Justin Kruger and David Dunning published a study reporting that people with low competence in a domain tended to overestimate their ability, while more competent people assessed themselves more accurately. It became the Dunning–Kruger effect, and then it became a way of calling people stupid on the internet, which is not what the paper says.

It is also contested. Later work has argued that much of the classic curve is a statistical artefact: plot self-assessment against measured ability with ordinary noise in the data and you get something shaped like the effect whether or not the underlying psychology is real. That criticism has force, and anyone leaning on this idea should know it exists.

For operational purposes, though, the disputed part is not the useful part. What survives the argument is simpler and much harder to dismiss: confidence and competence are produced by completely different processes. Confidence is generated internally, out of familiarity. Competence is only ever established externally, by test.

Familiarity feels like understanding. In a meeting, reported out loud, the two are indistinguishable. That is the whole problem.

A team that has run a system daily for four years has enormous familiarity with it. They know its moods, which alert to ignore, which step to do twice on a Monday. None of that is the same as knowing what the system depends on — because you can develop deep familiarity with the ten per cent of a system you touch and hold no knowledge at all of the ninety per cent you do not.

Two numbers for the same system

The clearest illustration of this is not a software story.

When the Rogers Commission investigated the loss of the Challenger in 1986, Richard Feynman asked engineers and managers at NASA to estimate the probability of a catastrophic failure. The engineers' estimates clustered around one in a hundred. Management's figure was closer to one in a hundred thousand — a difference of three orders of magnitude, inside one organisation, about one system, held simultaneously.

Nobody was lying. The engineers had built their number from component test data. Management had built theirs from a record of successful launches. Both were reasoning honestly from the evidence in front of them. Only one group had evidence that could have told them they were wrong.

That is the shape of the problem in a single image. The most confident number in the room is frequently the one furthest from anything that was measured, because confidence accumulates from the absence of visible failure — and the absence of visible failure is not evidence of a healthy system. It is just the absence of evidence.

The absence of visible failure is not evidence of a healthy system. It is just the absence of evidence.

The expertise a single success buys

The individual version of this shows up early in a career and never entirely stops.

Someone gets a result. They automate a report that used to take a day. They stand up a tool that works. They ship an integration and it holds. The success is real, and it is theirs, and it should count for something.

What is easy to miss is how little it establishes. One success is evidence that an approach worked once, at one scale, under one set of conditions — including every condition that happened not to be present. It says nothing at all about the absent ones, because absent conditions leave no trace.

So the mistake is usually not the one it looks like from outside. The person is rarely overrating their ability. They are overrating the generality of their evidence, which is a more forgivable error and a considerably more dangerous one.

The tell is specific, and you can check it in a conversation. Ask someone what they built and how, and they will be fluent. Ask what would have made it fail, and a person working from one data point has very little to say — not because they are concealing anything, but because a single success does not contain that information. Fluency about the path taken, silence about the boundary.

The shapes it takes are familiar. A tool chosen because it comfortably handled two hundred rows, deployed against two million. A pattern that worked cleanly with one integration, committed to across nine. "We did this at my last company," where the last company had a constraint nobody noticed was doing the load-bearing.

Seniority does not resolve any of this. It changes the blast radius. Someone junior generalising from one success writes a script. A founder or a technical lead generalising from one success commits the organisation.

The process that only exists in a pair of hands

Ask a manager whether their critical processes are documented and the answer is usually yes.

Ask what happens if the person who runs one of them is unreachable for three weeks, and the answer changes shape. It stops being a fact and becomes a plan. Someone would pick it up. It is mostly written down. We would work it out.

Both answers are sincere. The first is true at the level the manager touches the process — there is a document, it describes the steps. The second is true at the level the process runs, where the document covers the ordinary path and the person covers everything else: the exceptions, the client who needs it done differently, the step that gets skipped when the numbers look a certain way.

Key person risk is almost never described as key person risk by the people carrying it. It is described as reliability. That person is excellent. Things go smoothly. Nobody has had to think about it in years.

Which is precisely the condition under which nobody checks.

The spreadsheet everyone is sure about

The spreadsheet nobody else understands is the most common form this takes. We have written elsewhere about why replacing one often solves the wrong problem; this is a different failure, and it sits happily on a spreadsheet nobody intends to replace.

It works. It has worked every month for six years. That track record is read, understandably, as evidence that it is understood.

But a spreadsheet does not tell you when it is wrong. Nothing is thrown when a formula silently stops including a new row, when a lookup range was never extended, or when someone pasted values over a calculation in 2023 and the number has been static ever since. The output still looks like an output — right shape, right cell, right time of the month.

The confidence is not in the numbers. It is in the fact that nothing has ever appeared to go wrong. Those are different things, and only one of them has been tested.

The debt nobody believes is there

"We'll document it later" is not really a scheduling statement. It is a claim about how much of the reasoning is worth writing down, made at the exact moment that reasoning is most available and feels most obvious.

Six months on, the person who deferred it will genuinely not remember why the retry limit is three, or why that one integration bypasses the queue. The knowledge did not go into a backlog. It evaporated. And because nobody wrote down what was skipped, the organisation has no record that anything was skipped at all.

This is why technical debt registers are usually so short. They list the debt somebody remembered to declare. The debt that costs money is the debt nobody believes exists — which by definition is not on any list, and which will be discovered by whoever is on call the night it finally matters.

Knight Capital is the canonical version of this at speed. In August 2012, the firm deployed new trading software to its servers. The deployment was believed complete. One server out of eight had not received the update, and old code on that machine began behaving in a way nobody had reason to anticipate. The firm lost roughly $440 million in about forty-five minutes.

Nobody had a wrong belief about the market. They had a wrong belief about their own deployment, and no mechanism that would have told them.

Assumptions are cheap in a project and expensive in a system

The same failure mode governs how software gets specified.

A brief is a compressed account of an operation written by someone who understands part of it. That is not a criticism; it is structurally what a brief is. The person writing it is usually further from the daily work than the people doing it.

Every gap in that account gets filled with an assumption. Almost none get written down as assumptions — they get written down as requirements, which is how a guess acquires the authority of a decision.

An assumption during a project costs nothing. It is a sentence. The same assumption, once it is load-bearing in a delivered system, costs whatever it costs to unpick a design built on top of it. The price does not change because the assumption was reasonable. It changes because it was never marked as one.

When the tool supplies the confidence

There is a newer contributor to all of this, and it is worth naming directly rather than leaving implied.

A language model produces fluent, well-organised, authoritative output on demand. That is what it is for, and the output is frequently good. The issue is not quality. The issue is what it does to a signal organisations have relied on for a long time.

Fluency used to be weak evidence of understanding. Not proof, but correlated — because the usual route to speaking fluently about a system was spending time inside one. That correlation is now broken. A confident, well-structured account of your own architecture can be produced by someone who has never opened it, and so can a script that runs.

Which removes the traditional tell. You can no longer infer, from the quality of someone's explanation, that anybody did the work of understanding.

It also cuts inward. "I checked with the model" has quietly become a form of verification on a lot of teams, and it is not one — it is a second unexamined belief stacked on the first. A model will describe your system as you described it. It cannot tell you what you left out, because it does not know what you left out either.

And the ratio has changed. A team can now produce more system per unit of understanding than at any previous point in the industry. Overconfidence used to be rate-limited by how much software a small team could physically write. It is not any more.

The tool is not the problem here — it supplies genuine capability, and we build with it. The problem is that it supplies the feeling of competence at the same speed, and from the outside the two are indistinguishable. That is also why an agent taking actions in your operation is a harder governance question than a faster one.

Anyone can ship now

The sharpest version of this is not happening inside engineering teams. It is happening to people who were never in one.

It is now genuinely possible to produce a working website, or an internal tool, or a customer-facing form, without being able to read what was produced. That is a real expansion of capability and mostly a good one — more people can solve their own problems without waiting for a budget cycle. We are not interested in gatekeeping it.

The difficulty is that building is the part the tool compresses. Owning is not, and the gap between them is invisible on the day it works.

What is missing from an artefact like this is rarely features. Features are what got asked for. What is missing is everything nobody knew to ask for: where the secrets are stored, what happens when a field receives something unexpected, whether there are backups and whether anyone has restored one, what the permissions actually allow, what happens when the third-party API changes its response shape, and what the path is when the data model turns out to be wrong in month four.

None of that surfaces during the build, because none of it is a question the tool was asked. And the person best placed to notice the omission is the person who does not have the experience to know it is an omission — which is the whole mechanism of this article, arriving in a new place.

Two beliefs then compound. The owner cannot tell the difference between a site that works and a site that is safe to put customer records into. And now, for the first time, neither can the person who built it. This is a different situation from inheriting a system a model wrote for a developer, where at least someone in the chain could read the output.

It works is the weakest possible evidence about a system. It has also just become the easiest evidence in the world to obtain.

The difference between knowing and having checked

The practical fix is unglamorous, and it is not a mindset.

You cannot introspect your way out of this. Asking a team to be less confident does not work, because the confidence is not a character flaw — it is the honest output of long familiarity. What works is replacing the question. Not "do we understand this system," which is answered by feeling, but "when did we last demonstrate that we understand it, and to whom."

That second question has a date attached. It can be false. It is the only version of the question that produces information.

This is also, unremarkably, why we start engagements by spending paid time in an operation before proposing anything to build. Not as a philosophy — because a brief tells you what someone believes about their operation, and the only way to find out where that belief and the operation diverge is to go and look. Sometimes the answer is that the assumptions were sound. That is a useful result too, and it costs a fraction of finding out later.

The same instinct scales down to things you can do without anyone's help. Have someone who did not build a process run it from the documentation, and watch where they stop. Restore a backup rather than confirming the backup job succeeded. Ask two people independently what a system depends on and compare the lists. Every one of these converts a belief into a measurement, which is the only exchange rate that matters here.

One question worth asking

Pick the system your operation would least like to be without. Not the largest one — the one whose absence would be felt first.

Now ask how you know it works the way you think it does. Not whether it works. How you know.

If the answer is that it has always worked, you have a track record, which is worth something. If the answer is that someone checked, and you can say when, and what they found, you have evidence. Most organisations have far more of the first than they think, and far less of the second.

Then ask the narrower version, because it is the one that catches the most: how many different conditions has that evidence actually been collected under? A system proven once, at one scale, on one ordinary Tuesday, is proven for one ordinary Tuesday.

The expensive part was never the bug. It was the interval between when it became true and when anybody went looking.