🗃️ Prefix Cache (Prompt Reuse)

Agents and chat apps send the same long system prompt and tool list with every request. Without a cache, FLM reads that whole prompt again each time, and on a 14k-token prompt that is 10 to 60 seconds before the first word appears.

The prefix cache lets flm serve remember what it already read. When a new request starts with text FLM has seen before, FLM picks up from where it left off and only reads what is new.

(Time to first token, measured on an NPU2 Linux machine: the first number is a cold prompt, the second is the same prompt with the shared part cached.)

The other supported models gain the same way. Measured on shorter prompts (about 1 to 2 k tokens), a cold prompt against the same prompt with its shared start cached:

✅ Where it applies

Not covered: the single-turn models (hy-mt2-flash, qwen3vl-flash, gemma4e-flash: they take one request and pin their system prompt, so there is nothing to reuse), the developer-preview *-core backends, lfm3, and the speech, embedding and robotics models. They are unchanged.

It is on by default. Nothing to configure for a single chat or agent.

⚙️ Options

flm serve qwen3.5:4b --prefix-cache-mb 8192 --prefix-slots 16

When the first prompt is saved, the server logs what a prompt costs and the limits in force:

[FLM]  Prefix cache: up to 8 snapshots in 4096 MiB (a saved row costs 32 KiB, a snapshot's state 49 MiB)

Tuning --prefix-min-gain

Every saved prompt costs a copy of the model’s state out of the NPU and host memory, and when memory or slots run out it pushes an older prompt out. When a conversation grows by only a few tokens per turn, saving every turn would fill the cache with near-copies of each other. --prefix-min-gain N skips a save unless the new prompt adds at least N tokens over a saved prompt that already covers it. The server logs it when it skips:

[FLM]  Prefix cache: not saved, only 70 tokens past slot 3 (needs 128, --prefix-min-gain)

The price of a skip is that the next request resumes from the older saved prompt and has to read the difference again, which is always less than N tokens (about a second at most for N = 512 on a 2B model reading about 700 tokens/s).

Measured with 6 interleaved agents (own 2.5k-token system prompt, 10 turns each, tool results of 30 to 1500 tokens, plus title-style side calls) on qwen3.5:2b, 16 slots, two runs averaged. Turn time is relative to N = 64:

With fewer slots than conversations (8 slots for 6 agents here) the cache thrashes and N made no difference to the time. A synthetic workload on one model: measure your own agents (usage.prompt_tokens_details.cached_tokens and the time to first token, see below) before moving far from the default.

How much memory does it need?

A saved prompt costs a fixed amount per model plus an amount per token:

The other models:

(Measured from the server log for the models we ran; the 3B and 8B Llama and the 0.6B to 8B Qwen3 sizes follow from the model shape.) qwen3.8-mtp:27b keeps fewer and smaller defaults, 4 saved prompts in 1536 MiB, because each saved prompt is large; --prefix-slots and --prefix-cache-mb still override them.

Conversations that share a system prompt share its saved copy, so one agent with several sub-agents costs much less than the same number of unrelated conversations.

Measured with four unrelated 15k-token conversations taking turns and default options, the whole server process (model included) peaked at about 9 GB on qwen3.5:4b and about 12 GB on qwen3.5:9b.

Rules of thumb

• One agent, or a few agents sharing a system prompt: the defaults are enough.
• Several unrelated long conversations (different system prompts): each needs up to two saved prompts (its system + tools, and its latest turn). With the default 8 slots about four conversations stay cached; raise --prefix-slots (and --prefix-cache-mb) for more.

• If one prompt is bigger than --prefix-cache-mb, it is not saved and the server says so:

  [WARNING]  Prefix cache: a 140000-token prompt needs 4424 MiB, more than --prefix-cache-mb (4096 MiB); it is not saved. Raise --prefix-cache-mb to reuse prompts this long.
  

When a limit is reached, FLM drops the least useful saved prompt first: older replies of a conversation that has a newer one, then the least recently used reply, and only then a shared system-prompt copy that nothing depends on.

🔍 How to see that it works

The response reports how much of the prompt came from the cache:

"usage": {
  "prompt_tokens": 14046,
  "prompt_tokens_details": { "cached_tokens": 14012 }
}

The server log shows each decision:

[FLM]  No saved prefix matches, prefilling 13742 tokens          <- first request: nothing to reuse
[FLM]  Prefix slot 3: resuming at 9391 of 9408 tokens            <- a sub-agent reuses the shared system prompt
[FLM]  Prefix cache: dropping the least recently used end snapshot (9420 tokens)

🧠 How it works

1. While FLM reads a prompt it saves a snapshot of the model’s state at the end of the system prompt and tools, and another one at the end of the prompt.
2. For a new request, FLM looks for the longest saved snapshot whose text is exactly the beginning of the new prompt. It restores that snapshot and reads only the rest.
3. Snapshots are matched on the exact tokens, so two different conversations that start with the same system prompt reuse the same snapshot of it. Images are matched by their content: the same picture is reused, a different picture is read again (the text before it is still reused).
4. A short request with its own system prompt (for example the title call some agents send after each turn) does not erase the snapshots of the main conversation.

Real Hermes runs (official Docker image), average time to first token per request, stock FLM v1.0.10 against this cache:

In the multi-agent runs the first sub-agent reads the shared system prompt and its siblings reuse it, and the parent agent resumes its own earlier prompt after the sub-agents finish (Qwen3.6: 9.5 s instead of 61 s).

⚠️ Limits to know about

• The match is exact and from the start. Anything that changes the early part of the prompt (a timestamp in the system prompt, tools listed in a different order, an edited system message) breaks the match at that point and everything after it is read again. Keep the system prompt and tool list identical and in the same order between requests, and put anything that changes (dates, per-request data) at the end.
• A repeated identical request does not resume at its full length: FLM must read at least one token to produce an answer, so a retry resumes at the end of the system prompt + tools.
• Short side calls are not saved. A reply is only saved if it adds at least 64 tokens over a snapshot that already covers it, so short side requests cannot push out a long conversation. In exchange, the next turn of a very short chat may read up to 64 more tokens.
• Qwen3.6 after a tool loop. Qwen3.6’s chat template rewrites the earlier assistant messages (their reasoning blocks) as soon as a newer user message arrives. After a tool-using turn, a new user message can therefore only reuse the system prompt and the first user message, not the whole history. No prompt cache can do better with this template; without the prefix cache nothing is reused.
• Answers are not bit-for-bit identical to a cold run. Resuming from a snapshot gives the same quality, but with greedy decoding some answers differ in wording (for example the order of a list) from what the same prompt gives when read from scratch: about 1 in 5 on Qwen3.5 and about 1 in 3 on Qwen3.6 in our replay tests. The cache that came before it behaves the same way.
• Hermes sub-agents. Sub-agents reuse the parent’s system-prompt snapshot only when every task’s context field is empty. A per-task context is placed in the sub-agent’s system prompt, which makes each sub-agent a different prompt that is read from scratch. Larger models (4B and up) leave it empty in practice; qwen3.5:2b sometimes fills it in.
• Qwen3.8-27B keeps the shared system prompt only. A saved prompt of this model is large (about 150 MiB plus 84 KiB per token), so it saves the shared start of a conversation (system prompt + tools) and nothing after it, for text prompts. A conversation that keeps going is still continued from the end of its last turn with the older single-checkpoint reuse, as before; the saved system prompt is used when another conversation ran in between, so several agents that share one system prompt do not read it again each time.
• Gemma 3 has no shared system-prompt snapshot. Its chat template puts the system prompt inside the first user turn, so two conversations that differ in their first message have no text in common to save as a shared start. Each conversation still reuses its own earlier turns.
• Models with a sliding window or a convolution state (Gemma 3 and 4, gpt-oss, the LFM2 family) save a state that cannot be rewound, so a saved prompt is resumed only when the new prompt starts with all of it, and a prompt that shares just the first part of a saved one resumes at the saved system-prompt start instead.
• Pictures and audio. The Qwen-VL models, Qwen3.5 and MiniCPM-V reuse prompts that contain pictures. The Gemma 4 models (and the nemotron3.5-bolt model built on them) reuse text prompts through the prefix cache; a prompt with a picture or audio keeps the older single-checkpoint reuse. qwen3.8-mtp:27b is text only.
• Platforms. Tested on Linux with the XRT backend. The Windows and HRX builds contain the same code but have not been run.

🛠️ Troubleshooting