Edge Cases Are the Whole Job

Every impressive AI demo has the same secret: it was built to succeed. Of course it worked. It always works in the demo. The demo is the easy part, the sunlit ninety percent where the inputs are clean and the world behaves. The actual job (the reason this work is hard and the reason it’s worth paying for) lives entirely in the other part. The weird part. The edge cases. And I’ve collected some beauties.

The dog that threw a 90-mile-an-hour fastball

My favorite came from a sports-technology idea a team once walked me through. I was helping them figure it out, and it never became a real project, but I still think about it. The concept was an augmented-reality system meant to watch a pitch and call it a ball or a strike, tracking the ball through the air and judging where it crossed the plate. Clean idea. Great demo. Pitch comes in, system calls it, everyone claps.

Then you let it loose on the real world, and a white dog trots into the frame. Or a stray volleyball rolls through. And the system, which has been trained to find a small round object moving through its field of view, looks at this white dog and concludes, with total confidence, that someone has just thrown a ninety-mile-an-hour fastball made of dog. The whole thing falls apart not because the core idea was wrong but because the real world is full of round-ish white-ish things that are not baseballs, and every single one of them has to be accounted for.

That’s not a footnote. That is the job. Anyone can track a baseball against a clean background. The engineering (the real, hard, valuable work) is teaching the system what to ignore. The reference point I always reach for here is self-driving cars, which had to be painstakingly trained not to slam to a panicked halt every time something unexpected entered the road, because a car that freezes at every surprise is useless, and the world is nothing but surprises. Handling the surprise gracefully is the entire discipline. The happy path is free.

“The happy path is free.”

The demo that caught a live one

Sometimes the edge case cuts the other way and becomes the best sales moment you could never have scripted.

I know of a demonstration of a classification system (something built to scan a scene and flag objects of interest) being shown off, presumably with the expectation that it would identify some ordinary, pre-arranged examples. Instead, mid-demo, it flagged something real. It pointed at an actual object in the actual environment and classified it as a genuine item of concern, and it turned out to be correct: the flag led to a real-world action and, as I heard it, a follow-on contract. The demo caught a live one. Nobody planned it. The system did the thing it was built to do, in the wild, in front of exactly the right audience, and the accidental proof was worth more than any slide.

I love that story because it’s the inverse of the dog. Same lesson from the opposite direction: the interesting behavior is always at the edges. Sometimes the edge embarrasses you and sometimes it makes your quarter, but it’s never in the boring middle you rehearsed.

The hardest problem in the world is a pizza order

Here’s the one that keeps engineers humble. You’d think the frontier of conversational AI would be something lofty: legal reasoning, medical diagnosis. In my experience the truly brutal test is taking a pizza order.

Think about what actually happens when a person orders a pizza out loud. They change their mind mid-sentence. They say “no wait, half pepperoni. Actually make the whole thing pepperoni, but light on the cheese, and can you do the thing you did last time.” They negotiate with someone else in the room while still talking to you. They use a private shorthand and expect you to track it. They contradict themselves and don’t notice. A human order-taker handles this without thinking. For a machine it’s a nightmare of ambiguity, correction, context, and implied state: every hard problem in natural language, stacked into a sixty-second exchange about lunch. When someone tells me they’ve built a conversational agent, I don’t ask if it can pass some grand benchmark. I ask if it can take a chaotic pizza order from an indecisive human without falling apart. That’s the real bar, and it’s higher than it sounds.

Design for the tail on purpose

Once you accept that the tail of weird cases never fully goes away, the work becomes designing for it deliberately instead of pretending it away. And the single most valuable behavior you can build into one of these systems is the humility to know when it doesn’t know.

A system that recognizes it’s uncertain and hands off (routes to a human, asks a clarifying question, flags the case for review) is worth enormously more than one that plows ahead and gives you a confident answer to a question it didn’t understand. The confident wrong answer is the dog clocked at ninety miles an hour: fluent, sure of itself, and completely wrong. The honest “I’m not sure about this one, let me get a person” is the system saving you from a disaster you’d otherwise only discover after it shipped. So you build confidence thresholds, graceful fallbacks for the residue, and a human in the loop for the cases the machine flags. And you log every strange input, because today’s baffling edge case is tomorrow’s training example. The tail shrinks over time, but only if you’re capturing it instead of quietly swallowing it.

Let me admit something a little self-deprecating here, because it’s true of most of the best builders I know: a lot of us do this work because we’re intellectual out of pure laziness. I cannot stand doing tedious things by hand, and that intolerance is exactly what drives me to automate them properly, which means facing the edge cases honestly, because a half-built automation that fails on the weird inputs just hands the tedium back to me with extra steps. The laziness, pointed the right way, becomes rigor. You engineer the tail carefully precisely because you refuse to be the human fallback for your own sloppy system. The people who resent the edge cases build brittle things and spend their lives patching them. The people who fall in love with the edge cases build robust things and get to walk away. Same weird one percent. Completely different life.

Which is why you never promise 100%

All of this connects to the least glamorous and most important thing I do on any project: managing expectations about the edges up front.

Because the client, having seen the demo, naturally assumes the demo is the product: that the clean ninety percent is the whole thing and it’ll just work everywhere. My job is to gently kill that assumption before it kills the project. I say some version of: if we handle sixty percent of your cases automatically and reliably, that is a massive win. And it will not be one hundred percent, ever, so let’s design for that from the start. There will always be a tail of weird inputs, a residue of dog-shaped fastballs, that need a human or a fallback. Pretending otherwise doesn’t make the tail go away; it just guarantees the client feels betrayed when they meet it.

The teams that succeed are the ones that fall in love with the edges instead of resenting them. Because the edges aren’t the annoying exception to the work. They’re where the actual engineering is, where the real defensibility is, where the money is, and (if I’m honest about why I do this at all) where the fun is. Anyone can demo the baseball. Figuring out the dog is the whole job.