Good Enough Runs Local Now

There’s a default assumption baked into most conversations about AI right now: that “using AI” means calling out to one of a handful of giant frontier models, over the internet, paying by the token, forever. It’s such a common assumption that people don’t even notice they’re making it.

It’s also already out of date. For a large and growing share of what real businesses actually need, you don’t need the frontier at all. The model can run locally (on your own hardware, inside your own walls) and it’s good enough. That word, “good enough,” is doing a lot of work, and it’s worth sitting with, because it changes the economics and the risk profile of this whole thing.

”Good enough” is a real engineering standard, not an insult

Engineers sometimes hear “good enough” as settling. It isn’t. It’s the entire discipline. The question is never “what’s the most powerful model that exists?” It’s “what’s the least powerful model that solves this problem reliably?” because that’s the one that’s cheapest, fastest, and easiest to control.

And the smaller, open models you can run yourself have gotten shockingly good. A couple of years ago they were toys next to the frontier. Now, for summarizing, classifying, extracting, drafting, routing, answering questions against your own documents (the bread and butter of business AI), they hold their own. The frontier still wins on the genuinely hard stuff: novel reasoning, the really long chains, the tasks at the edge of what’s possible. But most business tasks are not at the edge of what’s possible. They’re mundane, and mundane runs local.

“They’re mundane, and mundane runs local.”

The routing insight

The way this actually gets built in practice is with a routing layer, and it’s one of my favorite ideas in the whole space because it quietly undoes the metered-frontier business model.

You put a lightweight dispatcher in front of everything. Every request comes in, and the router asks: how hard is this, really? The overwhelming majority (the routine questions, the simple transforms) get handled by a small local model for essentially nothing. Only the genuinely difficult requests, a small slice of the total, get escalated up to an expensive frontier model. When it’s tuned well, the fraction that ever needs to touch the pricey model is tiny.

Sit with what that means. If only a sliver of your traffic ever reaches the frontier, then the business model that assumes all of your traffic will pay frontier prices is in trouble. This is a big part of why I think the token-billing economics are unsustainable, not because the frontier isn’t valuable, but because most work doesn’t need it, and the tooling to route around it is getting better and more common by the month.

Why regulated clients go local first

For a lot of my clients, the local option isn’t a nice-to-have. It’s the only door they can walk through.

If you’re in healthcare, finance, or any field bound by compliance and certification, the idea of shipping your sensitive data off to a third party’s servers to be processed is a non-starter, or close to it. A model you host yourself sidesteps that entire category of objection. The data never leaves. You can point to exactly where it lives. And there’s a control argument that closes deals on its own: you hold all the keys. If you ever need to shut it off, you shut it off. You can physically unplug it. For a nervous compliance officer, “we can remote-disable this instantly and it never phones home” is worth more than any benchmark score.

There’s a cost angle too, and it’s a long-game one. Renting intelligence by the token is fine at small volume, but as usage grows, the meter grows with it, forever. Owning the hardware is a bigger number up front and a much smaller number over time. For an organization thinking in years rather than quarters, the math increasingly favors owning.

The paradox nobody mentions

Now, the honest complication, because I don’t like tidy stories that skip the hard part.

Everyone’s data already lives in the cloud. That’s the last twenty years of software. We all moved everything up there. So there’s a genuine paradox in “run your AI locally”: if the data you want to analyze is sitting in a dozen cloud services, then “local analysis” means pulling it back down to your own environment to work on it, which is its own project with its own costs and its own security questions. The clean picture of a self-contained local brain humming away in your server closet runs into the messy reality that the material it needs to think about is scattered across the internet under other people’s logins. Nobody has fully solved this. It’s one of the more interesting open problems in actually deploying this stuff, and anyone who waves it away hasn’t tried to build the thing.

How I actually place a client’s models

When I sit down to design this for a real business, the exercise is unglamorous and it comes down to one repeated question: what’s the least powerful model that solves each specific task reliably? You go through the actual work, piece by piece, and you place each piece where it belongs.

The routine, high-volume, low-stakes work (the summarizing, the tagging, the first-draft generation) goes to a small model, ideally local, where it costs almost nothing and you control it completely. Anything touching sensitive or regulated data goes local by default, because the compliance conversation is so much shorter when you can say the data never left the building. And the genuinely hard reasoning (the rare request that actually needs the frontier’s horsepower) gets routed up to a big model, but treated as a swappable dependency, never the foundation. You want to be able to change frontier providers the way you change electricity suppliers, without rearchitecting anything, because their prices and their terms will keep moving.

Two things I always warn clients about. First, budget for the data plumbing, because it’s the real cost and the real work: getting your scattered, messy information into a state where any model can use it is usually a bigger job than the model itself, and it’s the part the exciting demos never show. Second, don’t try to boil the ocean. Pick one workflow, place its models thoughtfully, measure whether it actually delivered, and expand from there. A modest hybrid setup that reliably handles one real process beats a grand architecture diagram that never survives contact with your actual data. The goal isn’t the most impressive stack. It’s the cheapest, most controllable stack that does the job, and increasingly, that stack has its center of gravity much closer to home than the default assumption would have you believe.

A couple of things that keep me honest

Two more field notes.

The models can now hold enormous amounts of text in mind at once: context windows big enough to swallow entire novels with room to spare. It’s genuinely impressive, and then you go to actually use it for a client and hit the deflating truth: most companies don’t have enough of their own organized, usable data to fill a fraction of that capacity. The bottleneck stopped being the model’s memory a while ago. It’s your own information being a disorganized mess. Which loops right back to the boring, essential work of getting your house in order before the fancy stuff can help.

And the choice of stack is rarely made on pure merit anyway. In practice, the model a company ends up using is heavily shaped by which vendors it’s already married to: the cloud ecosystem it’s already deep in tends to win, because that’s the path of least resistance for their procurement and their IT team. Technical superiority is one input. Existing relationships are often the louder one.

None of this dims my enthusiasm. It just points it somewhere specific. The future I’m building toward for clients isn’t “rent the biggest model and pray the bill stays sane.” It’s a hybrid: most of the work handled cheaply and privately, close to home, reaching out to the frontier only for the rare thing that truly needs it. Good enough, most of the time, turns out to be more than enough.