Why Your AI "Fine-Tuning" Budget Is a Total Waste of Capital in 2026

topics: AI, Technology, Strategy

We’ve spent the last few years obsessed with "owning" the model. Sold on the fear of hallucinations, companies were told that the only solution was to build custom weights. This was a convenient lie, as nothing actually prevents an LLM from hallucinating.

The Fine-tuning Fiasco

Research and recent statements from lead engineers at OpenAI and Anthropic suggest a troubling side effect: fine-tuning can actually "force" a model to be more confident in its errors. While a prompted model can be steered toward flexible "I don’t know" guardrails through multi-model evaluation systems, a fine-tuned model is often pressured into specific patterns. This makes its hallucinations harder to spot because they sound so certain. Fine-tuning, it turns out, is an architectural solution for ...nothing. More recently, fine-tuning is being promoted as a solution for making really tiny models take on some of the work of larger models. This is creative and interesting, but you have to be a real pessimist about the cost of inference in 10 years to spend any time on this.

This push for massive training and embedding projects became a corporate security blanket. These are high-cost, long-horizon projects with no guaranteed ROI, which, strangely, makes big-tech leadership feel more comfortable than giving teams all of these tools and asking only that they drive results. We've been lead to believe that innovation meant tinkering with the engine, while the real power was in how you steered the car.

Orchestration over Training

After a decade in the trenches of big-data Machine Learning, I find the sudden "re-discovery" of ML via LLMs almost boring. It’s not that the technology isn't impressive, it’s that we’ve finally reached the endgame. LLMs and machine vision models are, quite clearly, a significant apex of what this field can achieve. They are a much more powerful reality than anything we’ve been able to build in the last ten years. While I understand the excitement around "what else might we be able to build?," there seems to be a lack of stopping to smell the flowers. What we've built is actually sick AF. Let's use it.

The Art of Prompt Engineering

Perhaps it is the raw accessiblity of this new technology - that anyone, without any technical background, can do it, that makes it less appealing. Or perhaps it's the noisiness of it. LLMs do hallucinate and make mistakes, and it is a real new problem.

One of the most interesting architectures of prompt engineering is cascading, layered prompts and evaluation checks. You can add some external tools and sources to this layered architecture as well, but the real magic just happens when you use LLMs as an evaluator. Most businesses treat an LLM like a generator, or worse, a chatbot.

$Input -> Output$

This generates mistakes easily.

Robust systems use a chain of prompts.

Take a high-stakes field like Medical QA:

  1. Rather than one prompt asking "How do these drugs interact?", you use a cascade of 12 different evaluation prompts.

  2. You have prompts specifically designed to check for drug-to-drug contradictions, but instead of just asking "do these drugs interact? and how?" you have layered prompts that check for alternate names for those drugs, and then multiple permutations of prompts that check for specific drug to drug interactions you want to check for, with every possible combination of drug name you've identified.

  3. In this example, you are more worried about false negatives than false positives, so you would design the whole system to try to catch as many positives as possible, instead of trying to disprove them. Some systems maybe steered towards the opposite flow.

  4. The Result: Only after the "evaluator" prompts have signed off does the system generate a final result.

Turning Noise into Structure

The real market share is in the world's mountain of noisy, unstructured data. LLMs and machine vision models are the ultimate Noise-to-Structure engines. The massive opportunity today is using prompt engineering to turn messy reality like handwritten notes, chaotic emails, or legal discovery into tagged data. We need logic that can look at 500 pages of data and extract a perfectly structured JSON schema without breaking a sweat. Pro-tip: You don't send all 500 pages to an LLM at once.

The "Cost" Myth

Inference is the New Hosting.

If a 12-prompt evaluation chain sounds expensive to you, you aren't being optimistic enough about the trajectory of inference. We are seeing a race to the bottom in compute costs. Fine-tuning a small model to perform a single evaluation task is an inefficient use of human time when medium-sized general models already perform that task better and cheaper. While combining medium and large models is a "useful trick" for the present, your long-term strategy must assume that the cost of inference will eventually reach near zero—becoming as trivial as the cost of basic web hosting.

Missing the Trees for the Forest

The industry is currently obsessed with "Bigger Tech" like new MCP servers and massive fine-tuning clusters. In this rush for infrastructure, we are ignoring the most sophisticated tool already in our hands: The Prompt.

Prompt engineering isn't a "temporary hack" or a vibes-based workaround; it is the new software engineering. We currently lack the tooling for sophisticated prompt orchestration because the collective ego of the industry is still looking for a "bigger" technology to solve architectural problems. The real "Wow" moment of 2025 isn't that models got bigger; it’s that they became commodities. In many ways, the models are now interchangeable. While we still argue over which frontier model is "best" today, those marginal differences will effectively vanish by the end of 2026. At that point, the model will be a baseline utility, the "electricity" in the walls, while prompt orchestration will still be the emerging frontier where all the actual innovation happens.