Who's Liable When the Machine Is Confidently Wrong?

A regular piece of software fails in a way you can usually see. It throws an error, it crashes, it returns a blank. You know something went wrong because something visibly went wrong.

Language models don’t fail like that. They fail by being articulate. They hand you a confident, well-formatted, entirely wrong answer, in the same voice they use for the right ones, with no tremor to tell you which is which. That’s a genuinely new kind of failure, and our institutions have not caught up to it. We’re only just starting to fight about the question that matters most: when the machine is confidently wrong, who’s on the hook?

Hallucination is not a bug

Let’s get the framing right, because most of the public conversation gets it backwards. When one of these models makes something up, that isn’t a malfunction. It’s the technology doing exactly what it was built to do. These are probabilistic systems. They predict plausible next words. They are, in a very real sense, always making it up. It’s just that most of the time the most plausible thing to say also happens to be true. The “hallucination” isn’t a separate broken mode. It’s the same machinery, landing on a plausible falsehood instead of a plausible fact.

Once you internalize that, the error rate stops looking like something you’ll patch away and starts looking like a permanent property you have to design around. And even a low rate is a real problem at scale. A system that’s wrong five percent of the time sounds pretty good until you run a million interactions through it and remember that fifty thousand of them were confidently, fluently false, and that the humans on the other end had no reliable way to tell which fifty thousand.

”You should have checked” doesn’t hold

The industry’s favorite defense, when one of these mistakes causes real harm, is some version of the user should have verified. Read the fine print. Don’t rely on it for anything important. Always double-check.

I find that argument close to dishonest, and I think the courts are starting to as well. There was a ruling (I’ll keep it generic) where a major provider was held responsible for errors its AI-generated summaries produced. The reasoning was sharp: when the system generates an answer, it’s producing new content, not just pointing you at someone else’s. It’s not a neutral index handing you a list of links to evaluate. It’s an author. And authors are responsible for what they publish.

That cuts straight through the “just verify it” defense, and here’s why the defense was always hypocritical: these tools are marketed precisely as the thing that saves you from having to verify. The entire pitch is “skip the tedious research, we did it for you, here’s the answer.” You cannot sell a product as a trustworthy shortcut and then, the moment it’s wrong, turn around and say the user was a fool for trusting the shortcut. You don’t get to have it both ways. If the value proposition is “don’t bother checking,” then the liability for what you didn’t have to check is yours.

“You cannot sell a product as a trustworthy shortcut and then, the moment it’s wrong, turn around and say the user was a fool for trusting the shortcut.”

Where this bites in the real world

This isn’t abstract for anyone deploying AI in front of customers or in a regulated field, which is most of my clients.

Take legal research. The technology is genuinely useful there, until it invents a case citation that looks perfect and doesn’t exist. That failure mode is so well-documented now that using these tools in law without a human verification step is close to malpractice. The value is real, but it’s only unlockable with a human in the loop who owns the final answer. The AI drafts; the professional signs. The signature is where the liability lives, and it should be a human one.

The same logic applies anywhere the stakes are real. If you put an AI on your phone lines and it quotes a customer a price, a policy, or a promise, you made that promise. If it gives medical-adjacent or financial-adjacent guidance, you gave that guidance. The machine’s confidence doesn’t transfer the responsibility off of you and onto the vendor, and it certainly doesn’t transfer it onto the customer who trusted the tool you chose to deploy.

The confident-wrong tax compounds

Let me make the scale problem concrete, because it’s where good intentions quietly go bad. Say you put a customer-facing assistant into production and it’s right ninety-five percent of the time. That sounds excellent. In a small pilot it is excellent: a handful of interactions, you’d probably never even see the errors. So you roll it out.

Now run a million conversations through it. Five percent is fifty thousand confidently wrong answers, each one delivered in the same trustworthy voice as the right ones, each one a promise or a fact or a piece of guidance issued in your company’s name. Some of those become complaints. Some become refunds. A few, in the wrong domain, become lawsuits. And here’s the insidious part: because the tool is so fluent, nobody on your side catches the errors as they happen. There’s no crash, no error log, no red flag. The mistakes propagate silently and you find out about them from angry customers, weeks later, all at once.

That’s the confident-wrong tax, and it compounds precisely because the failures are invisible until they aren’t. Trust with customers erodes slowly and then breaks suddenly, and “the computer said so” has never once made a wronged customer feel better. So the answer isn’t to abandon the tool (the ninety-five percent is real value); it’s to instrument honestly for the five. Sample the outputs and actually read them. Put human review on the high-stakes categories where a wrong answer is expensive. Build the escalation path so the system hands off when it’s unsure instead of confabulating. You’re not trying to reach a hundred percent; you can’t. You’re trying to make sure the five percent lands somewhere you can catch it, rather than in a customer’s inbox with your logo on top.

Draw the lines in the contract

So what do you actually do about it? You stop pretending the risk isn’t there and you write it down.

When I help a business put one of these systems into production, we decide explicitly where it’s allowed to act on its own and where a human has to be in the loop. Anything with legal, financial, medical, or safety weight gets a human owner by design, not by hope. We keep the audit trail (transcripts, logs, a record of what the system said and when) because when something goes wrong, and eventually something will, “we have no idea what it told them” is the worst possible place to be standing.

And there are places I simply won’t go. Tools that build deep psychological profiles to influence people cross a line for me; the potential for manipulation is too high and the guardrails too flimsy. Anything that positions a bot as a substitute for real mental-health care is a hard no. Those aren’t legal judgments (the law hasn’t caught up); they’re just lines I’m not willing to be on the wrong side of when it does.

Because it will. The pattern with every powerful technology is the same: the capability arrives years before the accountability does, and there’s a messy window in between where the people deploying it get to decide whether they’re going to be responsible on their own or wait to be forced. I’d rather be early. The machine is going to sound sure of itself either way. Someone human has to be responsible for what it says, and I’d rather that someone be a person who chose to be than a defendant who didn’t.