[Issue]: `usage.prompt_tokens` on the OpenAI-compatible endpoints reports only the tokens prefilled on the current turn

[Javinator9889]

Problem Description

usage.prompt_tokens on the OpenAI-compatible endpoints reports only the tokens
prefilled on the current turn, not the size of the prompt. In any multi-turn
conversation the reported context usage collapses back to roughly the size of the
newest message on every request after the first.

Operating System

Ubuntu 26.04.1 LTS (kernel 7.0.0-31-generic)

CPU

AMD Ryzen™ AI 7 445

GPU

AMD Radeon™ 840M × 12

ROCm Version

1.0.1

Installation Method

Package manager

Installed ROCm Packages / Versions

Installed ROCm packages / versions

N/A

ROCm Component

No response

Steps to Reproduce

Command that fails

flm serve llama3.2:3b

then two sequential requests on the same conversation:

# turn 1
curl -s localhost:52625/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "llama3.2:3b",
  "messages": [{"role": "user", "content": "say alpha"}],
  "max_tokens": 16
}' | jq .usage

# turn 2 -- same conversation, with turn 1's reply appended
curl -s localhost:52625/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "llama3.2:3b",
  "messages": [
    {"role": "user", "content": "say alpha"},
    {"role": "assistant", "content": "alpha"},
    {"role": "user", "content": "say beta"}
  ],
  "max_tokens": 16
}' | jq .usage

Expected

prompt_tokens grows with the conversation, because it is defined as the number
of tokens in the input prompt. Under the OpenAI contract any cached portion is
counted inside prompt_tokens and reported separately in
prompt_tokens_details.cached_tokens.

Actual

Turn 2 reports a prompt_tokens roughly the size of the new message alone, so the
figure drops instead of growing.

Cause

The prompt cache matches the incoming prompt against token_history and erases the
matching prefix so only the new tail is prefilled. The count is then taken from the
already-trimmed vector, in AutoModel::_shared_insert
(src/common/AutoModel/automodel.cpp):

tokens.erase(tokens.begin(), tokens.begin() + skip_count);   // cached prefix dropped
...
meta_info.prompt_tokens = tokens.size();                      // only the NEW tokens

Qwen3_5_Omni::insert has its own copy of the same pattern.

The line was correct when written — at that point no prompt cache existed and
tokens really was the whole prompt. It was invalidated by 996d80e
("refactor: rewrite prompt cache management logic", 2026-05-20), which introduced
skip_count and the erase in the same hunk where it edited this line to drop a
stale + 1, without adding the skipped prefix back.

(Optional for Linux users) Output of rocminfo --support

rocminfo --support output

N/A

Additional Information

Impact

Agent front-ends that size the context from usage never see it fill up. OpenCode
in particular decides when to compact from this figure, so it never compacts, and
the session degrades into a loop of the context appearing to reset and being refed.
Every tool-call result is another turn, so the effect compounds quickly.

Related

• add current kv cache len (tokens) in the json output #533 — request to expose current KV cache length in the JSON output. Adjacent:
  prompt_tokens_details.cached_tokens partly answers that need.

Fix

Proposed in #729.