Why Your AI "Fine-Tuning" Budget Is a Total Waste of Capital in 2026

topics: AI, Technology, Strategy

We’ve spent the last year obsessed with fine-tuning models and RAG. Sold on the fear of hallucinations, companies were told that the only solution was to "fine-tune" models to their specific "company data" or set up a vector database with all of their information. This was a convenient misrepresentation, as nothing actually prevents an LLM from hallucinating.

The Fine-tuning Fiasco

Research and recent statements from lead engineers at OpenAI and Anthropic suggest a troubling side effect: fine-tuning can actually "force" a model to be more confident in its errors. While a prompted model will more willingly share low confidence, a fine-tuned model is often pressured into specific patterns. This makes its hallucinations harder to spot because they sound so certain. Fine-tuning, it turns out, is an architectural solution for ...nothing. More recently, fine-tuning is being promoted as a solution for making really tiny models take on some of the work of larger models. This is creative and interesting, but you have to be a real pessimist about the cost of inference in 10 years to spend any time on this.

RAG is an over-engineered solution to a simple memory/database store problem, which has easier, prompt engineering-centered solutions. Almost anyone considering RAG doesn't actually need a vector database - most people are working with 10 thousand documents or less. They just need to be properly tagged and a search function can give you the same results with 10% of the work.

This push for massive training and embedding projects became a corporate security blanket. These are high-cost, long-horizon projects with no guaranteed ROI, which, strangely, makes big-tech leadership feel more comfortable than giving teams all of these tools and asking only for results. We've been lead to believe that innovation meant tinkering with the engine, while the real power was in how you steered the car.

Orchestration over Training

After a decade in the trenches of big-data Machine Learning, I find the sudden "re-discovery" of ML via LLMs almost boring. It’s not that the technology isn't impressive, it’s that we’ve finally reached the endgame. LLMs and machine vision models are, quite clearly, a significant apex of what this field can achieve. They are now a much more powerful reality than anything we’ve been able to build in the last ten years. While I understand the excitement around "what else might we be able to build?," there seems to be a lack of stopping to smell the flowers. What we've built is actually sick AF. Let's use it.

The Art of Prompt Engineering

Perhaps it is the raw accessiblity of this new technology - that anyone, without any technical background, can do it, that makes it less appealing. Or perhaps it's the noisiness of it. LLMs do hallucinate and make mistakes, and it is a real new problem.

One of the most interesting architectures of prompt engineering is cascading, layered prompts and evaluation checks. You can add some external tools and sources to this layered architecture as well, but the real magic just happens when you use LLMs as an evaluator. Most businesses treat an LLM like a generator, or worse, a chatbot.

$Input -> Output$

This generates mistakes easily.

Robust systems use a chain of prompts.

Take a high-stakes field like Medical QA:

  1. Rather than one prompt asking "How do these drugs interact?", you use a cascade of 10+ different evaluation prompts.

  2. You have prompts specifically designed to check for drug-to-drug interactions, but instead of just asking "do these drugs interact? and how?" you have layered prompts that check for alternate names for those drugs, and then multiple permutations of prompts that check for specific drug to drug interactions you want to check for, with every possible combination of drug name you've identified.

  3. In this example, you are more worried about false negatives than false positives, so you would design the whole system to try to catch as many positives as possible. Other systems may be steered towards identifying false positives.

  4. The Result: Only after the cascade of "evaluator" prompts have signed off does the system generate a final result. That final result can also be re-vetted by rival models or with additional context.

Turning Noise into Structure

The real opportunity is in the world's mountain of noisy, unstructured data. LLMs and machine vision models are the ultimate Noise-to-Structure engines. The massive opportunity today is using prompt engineering to turn messy reality like handwritten notes, chaotic emails, or legal discovery into tagged data. We need logic that can look at 500 pages of data and extract a perfectly structured JSON schema without breaking a sweat. Pro-tip: You don't send all 500 pages to an LLM at once. That orchestration before and after the LLM - that's where the magic happens.

Complexity

The "Cost" Myth

Inference is the New Hosting.

If a 100-prompt evaluation chain sounds expensive (or slow) to you, you aren't being optimistic enough about the trajectory of inference. We are seeing a race to the bottom in compute costs. Fine-tuning a small model to perform a single evaluation task is an inefficient use of human time when medium-sized general models already perform that task better and cheaper. While combining medium and large models is a "useful trick" for the present, your long-term strategy must assume that the cost of inference will eventually reach near zero, becoming as trivial as the cost of basic web hosting.

Missing the Trees for the Forest

The industry is currently obsessed with "Bigger Tech." In this rush for infrastructure, we are ignoring the most sophisticated tool already in our hands: The Prompt.

Prompt engineering isn't a "temporary hack" or a vibes-based workaround; it is the new software engineering. We currently lack the tooling for sophisticated prompt orchestration because the collective ego of the industry is still looking for a "bigger" technology to solve all of our problems. The real "Wow" moment of 2025 is that models, in racing to the finish line, became commodities. Open-source commodities, at that! In many ways, the models are now interchangeable. While we still argue over which frontier model is "best" today, those marginal differences will effectively vanish by the end of 2026. At that point, the model will be a baseline utility, the "electricity" in the walls, while prompt orchestration will still be the emerging frontier where all the actual innovation happens.