Are Thinking Tokens and Input Tokens Interchangeable? An Experiment

I started this experiment with a really simple goal: get high-quality output with the lowest latency possible from a specific model, DeepSeek V4 Flash, which I had selected for its cost and quality.

I wasn't really thinking about thinking tokens or input tokens or how they behave differently when I started this experiment. One of the first things I noticed is that the less input tokens I had, the less latency there was. That was the first pattern I saw.

Some background on what I was working on: I wanted to cut the latency of the AI-generated questions in the Habit journal guided check-in. Latency matters for any AI product, but it matters a lot when people are checking in on their feelings. A delay between questions in a guided reflection turns into a lousy experience fast. I didn't want speed at the expense of quality. I wanted both. Without quality, it's pointless. Without speed, people disconnect.

I've been working on this prompt by running manual batch tests and I found that (generally) the fewer input tokens, the lower the latency. My best performing prompt (before today) was therefore very short - I was able to write something extremely concise that still passed the right information to generate the question I needed generated. This aligned with everything I saw online about how DeepSeek worked. However, my agent said something curious to me when running an audit on my prompts. It mentioned the thinking tokens that DeepSeek used and referred to them as part of the "input token budget," which is not how I thought of it. I had not actually done any work to try to reduce the amount of thinking DeepSeek did.

I tested turning thinking off before and the results were, of course, terrible. But thinking of input tokens as a "budget" made me think differently: I had never thought to replace the budget spent by thinking with more input tokens. What I found through this experiment is that they are in some way interchangeable. But not just that: input tokens are a bit "cheaper" than thinking tokens. So if you have any opportunity to know how to "think better" than your model, it might be more effective to think for your model entirely.

My findings

My short tidy prompt lost. The winning version was longer, clearer, but brought the wait down to under one second at the median, roughly 62% faster. All because I replaced thinking tokens with input tokens.

The lever was spending that "budget" on more input tokens: a clearer, more explicit instruction for how to write a good question. And it worked! Quality went up too.

I'm not suggesting we "turn thinking off everywhere." However, this is a situation where if you already know what the model should be thinking, maybe you should think for it by turning thinking off & get better results. In a lot of enterprise workflows, this may be the better option. Even in enterprise workflows where contexts change, it might be worth having a dynamic instruction/thinking component that swaps out depending on the business context over using a model's native thinking functionality. That is - if latency matters.

The experiment

I tested seven variants on real user data (my own lol). I ended up with 175 candidate questions in total.

I judged outputs in blinded groups on four things:

• whether the question gave the person something useful to reflect on;
• whether it sounded simple and conversational;
• whether it stayed open rather than forcing a binary choice; and
• whether it naturally followed the earlier answers.

I also flagged the things I actively did not want: backwards, trick-question phrasing; an implied either/or; unsupported assumptions; clinical or poetic language; and repetition. Finally, I recorded time to the visible question.

Here are the results:

* Total tokens includes the visible output.

The winner had a longer system prompt than the production version. Yet it cut median latency from 2.26 seconds to 0.87 seconds and improved the quality score from 6.32 to 8.03.

The questions also became more recognizably human. Instead of "When you feel that spark of motivation, what else stirs inside you?", the better prompt produced questions like "What would something meaningful look like in your day-to-day life?" That is a small difference in wording, but a large difference in whether someone can answer without being annoyed AF at the phrasing first.

Are thinking tokens part of your input?

In my thinking-on runs, latency tracked reasoning tokens extraordinarily closely. Count the thinking tokens and the latency becomes almost predictable. The fastest (non-thinking) prompt was longer, 643 input tokens, more than the entire old request, and it returned in an average of 0.87 seconds. Less than half the time.

This gave me a more useful mental model than "tokens are bad." Every token is a potential latency cost, but not all tokens are the same cost in the same request. For a short, warm request like mine, the variable reasoning trace was extra expensive.

The deeper realization, for me, was that thinking tokens are costing you part of your "input budget." However, they are not interchangeable (thinking tokens are more expensive in terms of latency). A model with thinking on spends part of its budget privately working out what a good output looks like. A model with thinking off can reach the same place (sometimes a better one) if you do some of that thinking for it in the prompt. The reasoning has to happen somewhere; the only question is whether it happens as invisible, variable-cost deliberation at request time, or as explicit, one-time editorial judgment you write down yourself. For a task where I already knew what "good" sounded like, writing it down worked better.

Temperature was not a footnote

There was another variable hiding in plain sight. DeepSeek's documentation says that temperature and top-p settings are ignored in its thinking mode. Once I switched thinking off, temperature became active again. The same expanded prompt at a temperature of 1.3 scored 7.56 and had a 20% flag rate. At 0.7, it scored 8.03 with a 4% flag rate. We don't talk about temperature much anymore, but I find it a very good lever to adjust in contexts where creative writing matters.

For short prompts, caching doesn't matter

I also tested input context. The guided check-in resends the earlier question-and-answer history with each request, and DeepSeek automatically caches matching prompt prefixes. About three quarters of my prompt was already warm on later questions. I reshaped the conversation to push the cache-hit ratio from roughly 75% to 81%. The latency gain: nothing meaningful.

So the finding is simple. For short prompts like mine, prompt caching is not the lever. The cache was already doing its job, and the hidden reasoning trace was the thing dominating the clock. On a much longer context or a colder cache this could flip; DeepSeek's context-caching guide explains when it helps. But if your requests are short, don't bother tuning the cache. "Thinking on" may be a better place to start looking for optimization.

Why benchmarks can be true and still not answer your product question

Benchmarks are valuable. They give us common tasks, fixed procedures, and a way to see broad capability. They have to hold many variables constant; otherwise comparisons become impossible.

When you're building products, you often have very specific requirements and boundaries and goals that you are aiming for. And sometimes the benchmarks don't point you in the right direction. You actually have to get in there and try your workflow multiple times with every version of the same model, including temperature, including thinking on/off, and including a lot of variations on the system and user prompts.

The slightly counterintuitive ending is that my longer prompt was not bloat. It was a compact specification for the task.

That is the broader lesson I am taking with me: when an AI feature is slow or oddly mediocre, do not only ask, "Which model should I use?" Ask, "What version of this model am I actually testing? What levers can I tweak to get the result I want?"