Auto-Tune: Stop Writing the Prompt, Train the Model

TL;DR:

Your system prompt is a workaround for a model that was never trained on your job. The fix is a checkbox that nobody has shipped, and it's the largest unclaimed opportunity in AI infrastructure right now.

Nearly every production AI deployment I've worked on pays a general-purpose frontier model to do a narrow, repetitive job, then compensates for the mismatch by writing increasingly long instructions. What those teams actually need is a smaller model trained on the work itself, automatically, continuously, off the traffic already moving through the system. I want a flag on an API key that does this. Not a platform. Not a pipeline. A flag. I call it auto-tune, and I think it's where this entire industry ends up.

The industry is currently walking in the opposite direction. In May, OpenAI started emailing developers to say the fine-tuning platform was going away. Organizations that hadn't already run a training job were cut off immediately. The ones that had until January 6th, 2027, before they can't create new jobs either, though existing fine-tuned models keep serving inference requests until their base models retire. OpenAI's stated reason: the newest models have become capable enough to make most fine-tuning unnecessary.

I understand that argument. I think it's wrong, and the last two years of my job are the reason why.

I run Forward Deployed Engineering at CopilotKit, which means I spend my time inside other people's agent deployments. Seed-stage startups through the Fortune 500, in production, under load, with real money attached. You see things from that seat that never show up in a benchmark.

What the prompt actually looks like

I can't name customers. I can tell you what I open when I get access to a repository.

System prompts running to several thousand tokens are routine. Tens of thousands is common. I've more than once seen prompts in the hundreds of thousands of tokens: carefully assembled, heavily commented, versioned, argued over in code review, defended by good engineers who arrived there honestly. Nobody sets out to write a hundred thousand tokens of instructions. They write a thousand, find an edge case, write two hundred more, find another, and run that loop until the loop becomes the job.

This isn't incompetence. It's the only lever the platform gave them.

When a model gets something wrong, and your options are to prompt it differently or accept the error, prompt it differently. Every individual addition is justified. The aggregate is a document no human reads end to end and, as it turns out, no model does either.

The ceiling nobody tells you about

Here's what makes prompt engineering a dead end rather than merely a chore. The instructions stop working because there are a lot of them.

The effect has been documented since Liu et al.'s "Lost in the Middle" in 2023. Model attention across a long context is U-shaped: strong at the beginning and the end, 20 to 30 points weaker in the middle. It shows up in both base and instruction-tuned models, so it isn't an artifact of alignment. Chroma tested 18 frontier models and found that each degraded as input length grew. The million-token context window on your invoice is real. The million tokens of reliable instruction-following are a fiction.

The sharpest measurement I've seen is a benchmark called IFScale, which runs the obvious experiment nobody had bothered to run: hold the task constant and scale the number of instructions from ten to five hundred. At maximum density, the best frontier models manage 68% adherence. They also show a consistent primacy bias, meaning whatever you put first gets followed best, and that bias converges as density climbs. So the ordering tricks stop paying at precisely the density where teams have started depending on them.

Read that back against a hundred-thousand-token prompt. Your instructions aren't being ignored because they're badly written. They're being ignored because they're instruction number 380.

We tell our customers to break this up with subagents, and that's the right advice. It's what I'd tell you today. But it's a mitigation rather than a cure, because the subagents are still general-purpose frontier models whose behavior must still be specified in prose. You've turned one document nobody reads into six documents nobody reads, and you're still paying OpenAI's flagship rate for a component whose entire job is deciding whether an invoice is a duplicate.

Where OpenAI has a point

The wind-down email isn't foolish, and pretending otherwise would be cheap.

Frontier models genuinely did absorb a great deal of what fine-tuning used to buy. In 2022, you fine-tuned a model to produce reliable JSON. Today you just ask. A meaningful share of historical fine-tuning jobs were solving problems that no longer exist, and maintaining a platform for them is a poor use of anyone's engineering time.

There's a second objection, and it's the better one. A model trained continuously on its own production traffic is a model being trained on its own mistakes. Do that carelessly, and you get something confidently, fluently, expensively wrong, with the errors compiled into the weights instead of sitting in a prompt where you can find them and delete them. Any serious version of what I'm proposing needs graded outputs and a real evaluation gate behind it, not a firehose of logs.

Both of these are arguments about how fine-tuning has been packaged. Neither one establishes that a general-purpose model is the right engine for a narrow job.

The tooling is aimed at the wrong person

Consider what already exists.

Mira Murati's Thinking Machines shipped Tinker last October, an API for fine-tuning open models with LoRA across distributed GPUs, handling scheduling and crash recovery while leaving you in control of the training loop. It's a genuinely good product. It's also unmistakably built for someone who wants control over a training loop. That person is a researcher. That person is not the team shipping a claims-triage agent on a deadline.

OpenPipe turned production logs into distilled specialist models and was acquired by CoreWeave last September. Portkey has a private beta for a feature called Autonomous Fine-tuning, in which the gateway collects your traffic and you can schedule retraining daily, weekly, or monthly. Every component of what I'm describing exists somewhere, shipped and working.

And not one of the deployments I walk into has any of it switched on.

The reason isn't ignorance. All of it is a product you go adopt: a vendor to evaluate, a pipeline to own, a headcount to justify, a private beta to email about. It's never a default you inherit. It's never already on.

What auto-tune is

A boolean on an API key or a project ID that says: train on what I'm already doing.

The inputs, outputs, tool calls, corrections, and retries. All of it is already flowing through the provider's infrastructure and already being logged for billing and abuse monitoring. The training data has been sitting there the whole time. The only missing piece is somebody deciding that using it should be the default.

And it has to be continuous, which is the part I care about most. Not a job you run in March against a model you then freeze until someone files a bug. A model that goes from A to B, gets A/B tested against the version it replaced on live traffic, wins or loses on the numbers, then does it again. B to C. C to D. What makes fine-tuning feel like a research project today is that it's an event. Make it a process, and it stops being one.

The mechanism is already published. Thinking Machines' own work on on-policy distillation, where the student model generates and the teacher grades every token so the student learns exactly where it went wrong, reached reinforcement-learning-grade results at nine to thirty times better compute efficiency. One reasoning run hit its target in 1,800 GPU-hours, compared to 17,920 for the conventional approach. A large model teaching a small one isn't speculative. It's a paper with numbers, published by the same company that sells the researcher-facing version of the tool.

I don't need a model that can pass AP Physics. Some people do, and for them frontier pricing is fair, and the capability is worth every cent. What I need is a model that's exceptional at the four things my product actually does. That's a different purchase, and right now the market only sells the first one.

What it becomes

Follow this out and the destination isn't cheaper inference. It's a different architecture.

You want a delegator, a model whose only job is knowing which specialist handles this request, sitting above a fleet of small, sharp, independently trained models. Routing itself is old news. RouteLLM demonstrated 85% cost reduction at 95% of frontier quality on MT-Bench, and Martian and Not Diamond have built businesses on it. But every router shipping today routes between models somebody else trained. None of them route to a fleet you tuned yourself, on your own traffic, for your own work.

Someone will read this and say I've described mixture-of-experts, which nearly every frontier model has used for years. So let me be exact about the difference, because it's the whole argument.

MoE routes tokens to experts frozen inside a single training run. Every expert was trained together, ships together, updates together. When the model gets retrained, it gets retrained in its entirety, at enormous cost, on somebody else's schedule. What I'm describing routes tasks to specialists that retrain independently, on their own cadences.

That decoupling is the thing.

A model handling news needs retraining constantly. A physics-handling model is fine for a quarter. A model classifying your invoices needs retraining when your invoice formats change and never otherwise. Bind all three into one artifact, and you pay the cost of the fastest-moving component across every parameter in the system, forever. Separate them and each one updates exactly as often as its own domain actually moves.

I think this is where the frontier labs themselves land: an orchestrator over a population of specialists, each on its own clock, rather than one enormous artifact retrained wholesale. The economics are too lopsided for it to go any other way.

It shouldn't only be them, though. The providers have every incentive to keep you on the general-purpose model, and winding down a fine-tuning platform is exactly what that incentive looks like from the outside. This capability belongs in every gateway, every agent framework, and every inference provider that isn't also selling you the expensive model.

We spent two years teaching enormous general models to act like small specialized ones, one instruction at a time, until the instructions no longer fit. We have the distillation research. We have the routing research. We have the logs.

What's missing is a checkbox.

David McKay is Head of Forward Deployed Engineering at CopilotKit.