Reducing Latency: How Our AI Questions Went From 3 Seconds to Under 1

I had a slightly embarrassing assumption about AI latency: shorter prompts must be faster prompts.

It is a tidy idea. Fewer words in, fewer milliseconds out. It also turns out to be incomplete enough to send a product team in the wrong direction.

We were tuning the questions in Habit's guided check-in, the short, adaptive flow that helps someone reflect without facing a blank page. Each question needs to sound ordinary, follow what the person has already said, and arrive quickly enough that the check-in still feels like a fluid interaction rather than a tiny meeting with a loading spinner. I expected our shortest instruction to win. It did not. The winning version was longer, clearer, and brought the wait down from seconds to under one second at the median — roughly 62% faster than our production setup.

The lever was not magical prompt prose. We turned off DeepSeek V4 Flash's thinking mode, then spent some of the token budget on a more explicit instruction for how to write a good question.

For this small task, quality also went up.

That does not mean "turn thinking off everywhere." It means a model benchmark is only one view of a model. The product behavior you get depends on the task, the configuration, the prompt, the context shape, the temperature, and what you choose to measure.

The experiment: a better next question, quickly

We tested seven prompt-and-configuration variants on five real, de-identified check-in prefixes. For each prefix, we generated questions for stages three through seven of the flow: 175 candidate questions in total. The raw reflections never appear here; we used them only to make sure the prompts met the messy reality of an actual product rather than a polished demo.

We judged outputs in blinded groups on four things:

• whether the question gave the person something useful to reflect on;
• whether it sounded simple and conversational;
• whether it stayed open rather than forcing a binary choice; and
• whether it naturally followed the earlier answers.

We also flagged the things we actively did not want: backwards, trick-question phrasing; an implied either/or; unsupported assumptions; clinical or poetic language; and repetition. Finally, we recorded time to the visible question, rather than pretending an average response time is the whole user experience.

This was an exploratory product evaluation, not a universal benchmark. Each condition had one generation per scenario, so the useful conclusion is about what to test next and what to ship behind monitoring, not a claim that we have discovered a law of language models.

Here is the concise version of what happened:

The headline comparison is deliberately a little unfair to my original intuition: the winner had a longer system prompt than the production version. Yet it cut median latency from 2.26 seconds to 0.87 seconds and improved the quality score from 6.32 to 8.03.

The questions also became more recognizably human. Instead of "When you feel that spark of motivation, what else stirs inside you?", the better prompt produced questions like "What would something meaningful look like in your day-to-day life?" That is a small difference in wording, but a large difference in whether someone can answer without translating the question first.

The hidden token bill was thinking

DeepSeek V4 Flash has thinking enabled by default. In that mode, it can spend tokens reasoning before it produces the short, visible question. Those reasoning tokens are useful for problems that genuinely need multi-step work. A one-sentence follow-up question is usually not one of them.

In our thinking-on runs, latency tracked reasoning tokens extraordinarily closely. In the compact thinking variant, the median request used about 483 reasoning tokens and took 6.35 seconds at the median. In the winning no-thinking configuration, reasoning tokens were zero and the median was under a second.

This gave us a more useful mental model than "tokens are bad." Every token is a potential latency cost, but not all tokens are the same cost in the same request. For a short, warm request like ours, the variable reasoning trace was the expensive and unpredictable part. Turning it off removed a hidden, variable token budget. We could use a few more input tokens to tell the model plainly what good sounded like:

• ask one thing, in ordinary language;
• stay connected to the person's earlier answers;
• do not reverse a question into a small logic puzzle;
• do not write an either/or question when life plainly has more than two options; and
• move the reflection forward.

That is the trade that mattered: less private deliberation, more visible editorial direction.

The deeper realization, for me, was that thinking tokens and input tokens are interchangeable. A model with thinking on spends part of its budget privately working out what a good question looks like. A model with thinking off can reach the same place — sometimes a better one — if you do some of that thinking for it in the prompt. The reasoning has to happen somewhere; the only question is whether it happens as invisible, variable-cost deliberation at request time, or as explicit, one-time editorial judgment you write down yourself. For a task where we already knew what "good" sounded like, writing it down was the better deal: the model stopped re-deriving our editorial standards on every request, and the answers got faster and more consistent at once.

Temperature was not a footnote

There was another variable hiding in plain sight. DeepSeek's documentation says that temperature and top-p settings are ignored in its thinking mode. Once we switched thinking off, temperature became active again. The same expanded prompt at a temperature of 1.3 scored 7.56 and had a 20% flag rate. At 0.7, it scored 8.03 with a 4% flag rate.

In other words, we had not merely changed the prompt. We had changed which controls worked.

This is exactly the kind of thing a one-line benchmark comparison will miss. If a team tests one model with thinking on, another with it off, and both at a nominal temperature that only one of them honors, it is no longer testing "the models." It is testing two different products configured in two different ways.

Context tokens still matter. Just not in the way I expected here.

There is a second, more familiar token story: input context. Our guided check-in sends the earlier question-and-answer history with each new request, so question four can build on questions one through three.

DeepSeek automatically caches matching prompt prefixes and reports cache-hit and cache-miss tokens. In our live-shaped requests, about three quarters of the prompt was already warm on later questions. We tried reshaping the same conversation to get an even better cache-hit ratio, from roughly 75% to 81–82%. It was a sensible experiment. It did not produce a meaningful latency gain for these short requests.

Why? The cache was already doing most of the useful work, while the reasoning trace was still dominating the clock. On a much longer context, or on a colder cache, the conclusion could reverse. DeepSeek itself says its context cache can significantly reduce first-token latency for long, repetitive inputs, and exposes cache-hit versus cache-miss token counts so teams can see what is happening. DeepSeek's context-caching guide is worth reading before trying to optimize a long-running chat or agent.

So the practical rule is not "make every prompt shorter." It is: measure the tokens you are actually paying for, including uncached input, visible output, and hidden reasoning. Then find the expensive part of your interaction.

This is bigger than DeepSeek

The mechanisms are provider-specific, but the product lesson is not.

• DeepSeek: V4's thinking-mode documentation describes thinking as enabled by default and notes that sampling controls such as temperature are ignored while it is on.
• OpenAI: its API exposes reasoning effort and reports reasoning tokens separately. The docs explicitly say that reducing reasoning effort can produce faster responses and fewer reasoning tokens. See the API reference.
• Google: Gemini's thinking configuration includes a budget control, including zero for Gemini 2.5 Flash, and Google frames it as a choice for applications with latency constraints. See Google's thinking guide.
• Anthropic: Claude's extended-thinking guide advises increasing the budget only as needed because higher budgets trade quality for latency. It also notes that thinking is not compatible with normal temperature adjustments. See Anthropic's guide.

This is not a contrarian discovery that nobody else has noticed. The providers document the knobs because the trade-off is fundamental. Researchers are now making it explicit too: THINK-Bench evaluates reasoning efficiency alongside answer quality, and HRBench studies when hybrid reasoning models should switch thinking modes. Even the DeepSeek V4 Flash community has been asking which sampling parameters are needed to reproduce reported results, a healthy reminder that a score without its configuration is not very reproducible. That discussion is public on the model card.

The useful novelty for us was applying that idea to a tiny, human-facing product moment. A two-second pause is not an abstract efficiency penalty when someone is trying to check in with themselves. It changes how the whole experience feels.

Why benchmarks can be true and still not answer your product question

Benchmarks are valuable. They give us common tasks, fixed procedures, and a way to see broad capability. They have to hold many variables constant; otherwise comparisons become impossible.

But that strength is also their boundary.

A benchmark typically tells you something like: "Under this prompt format, sampling configuration, tool setup, evaluation set, and scoring rule, model A got this result." It cannot automatically tell you: "Which configuration will give your users the best interaction under a one-second latency target?"

The gap gets wider for reasoning models because the settings can alter both the path and the answer:

1. Thinking or reasoning effort changes the amount of work a model may do before it responds.
2. Temperature and other sampling controls may be honored differently, or not at all, across modes.
3. Prompt structure and caching change how much context must be processed afresh.
4. The measurement itself changes the winner. Mean completion time, time to first visible text, total token cost, rubric quality, and failure rate are all legitimate metrics, but they reward different behavior.

That does not make benchmarks misleading. It means we should stop treating a leaderboard position as an instruction manual.

For a product manager, this is liberating. You do not need to win an argument about which model is "best." You need to construct a small decision surface for the interaction you are building: perhaps reasoning on/off × two temperatures × two prompt styles × three realistic task slices. Then decide which point gives you the quality, speed, and cost you actually need.

A practical testing protocol

If I were starting this experiment again, I would do the following before changing production:

1. Name the user-facing job. "Generate a helpful next question in under 1.5 seconds" is more useful than "make the model good at check-ins."
2. Test the reasoning setting first. On, off, or a few effort levels. Treat it as a product decision, not an implementation default.
3. Re-test sampling after changing modes. A temperature setting can be inert in one mode and meaningful in another.
4. Use real task shapes. Preserve privacy, but test the cumulative context, incomplete answers, ambiguity, and repetition a real user will create.
5. Log token categories, not just total tokens. Track uncached input, cached input, visible output, and reasoning tokens wherever your provider exposes them.
6. Score the failure modes your users notice. A technically correct question can still be awkward, leading, or impossible to answer.
7. Repeat the promising cells. Our experiment found a strong direction, but a production change deserves multiple runs, a holdout set, and monitoring after release.

The slightly counterintuitive ending is that our longer prompt was not bloat. It was a compact specification for a task that did not need a private essay before answering. We traded hidden computation for visible intent — reasoning tokens for input tokens — and got speed and quality in the same move.

That is the broader lesson I am taking with me: when an AI feature is slow or oddly mediocre, do not only ask, "Which model should we use?" Ask, "What version of this model are we actually testing?"

Sometimes the fastest route to a better answer is not less prompting. It is fewer invisible thoughts and a clearer brief.