Prerequisites
Please answer the following questions for yourself before submitting an issue.
Expected Behavior
Tokenization followed by detokenization should reproduce the original byte sequence exactly.
In particular, runs of whitespace that are successfully represented by tokenizer token IDs should not be discarded during detokenization.
For example:
tokens = llm.tokenize(data, add_bos=False)
restored = llm.detokenize(tokens)
assert restored == data
# Current Behavior
Long runs of spaces are tokenized successfully, but some of the resulting whitespace token IDs are detokenized to an empty byte string (`b''`), causing the whitespace to be lost.
For example, with the provided test:
- 32 leading spaces round-trip correctly.
- 40 leading spaces fail.
- 64 leading spaces fail.
- 128 leading spaces fail.
For 64 spaces, tokenization produces a whitespace token followed by the token containing the final space and `X`:
```text
tokens=[15270, 1599]
# Environment and Context
Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.
* Physical (or virtual) hardware you are using, e.g. for Linux:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 28
On-line CPU(s) list: 0-27
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i7-14700F
CPU family: 6
Model: 183
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 1
Stepping: 1
CPU(s) scaling MHz: 18%
CPU max MHz: 5400,0000
CPU min MHz: 800,0000
BogoMIPS: 4224,00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush
dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_ts
c art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf
tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma c
x16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsav
e avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp
ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1
avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsav
eopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat
pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku osp
ke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize
arch_lbr ibt flush_l1d arch_capabilities
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 768 KiB (20 instances)
L1i: 1 MiB (20 instances)
L2: 28 MiB (11 instances)
L3: 33 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-27
Vulnerabilities:
Gather data sampling: Not affected
Ghostwrite: Not affected
Indirect target selection: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Old microcode: Not affected
Reg file data sampling: Mitigation; Clear Register File
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence
; BHI BHI_DIS_S
Srbds: Not affected
Tsa: Not affected
Tsx async abort: Not affected
Vmscape: Mitigation; IBPB before exit to userspace
* Operating System, e.g. for Linux:
Linux mtor-xub-main 7.0.0-30-generic #30~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 7 13:27:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
* SDK version, e.g. for Linux:Python 3.12.3
GNU Make 4.3
g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
$
# Failure Information (for bugs)
### Failure Information
The failure occurs during detokenization, not tokenization.
The tokenizer preserves the long whitespace runs as token IDs, but some of those token IDs produce an empty byte string when passed to `detokenize()`.
Example with 64 leading spaces followed by `X`:
```text
Input:
b' X'
Token IDs:
[15270, 1599]
Individual token detokenization:
15270 -> b''
1599 -> b' X'
Expected:
b' X'
Actual:
b' X'
A systematic test gives:
1 spaces: OK
2 spaces: OK
4 spaces: OK
8 spaces: OK
16 spaces: OK
32 spaces: OK
64 spaces: FAIL
128 spaces: FAIL
# Steps to Reproduce
Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.
## 1. Create a virtual environment
```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
## 2 install llama-cpp-python
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python
## 3 Obtain GGUF model
I use ollama to install the model and then obtain the model path using:
ollama show qwen2.5:0.5b --modelfile | grep '^FROM'
Then I use that model ID in the script variable MODEL_PATH.
For reference, I have reproduced the same issue with smollm:135m, so the problem does not appear to be specific to the Qwen2.5 model.
This is the script to reproduce the issue:
```python
#!/usr/bin/env python3
from llama_cpp import Llama
MODEL_PATH = "/usr/share/ollama/.ollama/models/blobs/sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc870eda801d3ab0515"
TEST = (
b" 1\n"
b" 2\n"
b" 3\n"
b" 4\n"
b" 8\n"
b" 16\n"
b" 32\n"
b" 64\n"
b" 128\n"
)
llm = Llama(
model_path=MODEL_PATH,
n_ctx=512,
n_gpu_layers=0, # tokenizer test only; GPU irrelevant
verbose=False,
)
print("Original bytes:")
print(repr(TEST))
print()
tokens = llm.tokenize(TEST, add_bos=False)
print(f"Token count: {len(tokens)}")
print(f"Tokens: {tokens}")
print()
restored = llm.detokenize(tokens)
print("Restored bytes:")
print(repr(restored))
print()
if restored == TEST:
print("ROUND TRIP OK")
else:
print("ROUND TRIP FAILED")
# First differing byte
limit = min(len(TEST), len(restored))
first_diff = None
for i in range(limit):
if TEST[i] != restored[i]:
first_diff = i
break
if first_diff is None and len(TEST) != len(restored):
first_diff = limit
print(f"First differing byte offset: {first_diff}")
print()
start = max(0, first_diff - 80)
end_original = min(len(TEST), first_diff + 160)
end_restored = min(len(restored), first_diff + 160)
print("Original around mismatch:")
print(repr(TEST[start:end_original]))
print()
print("Restored around mismatch:")
print(repr(restored[start:end_restored]))
print()
print("\nTOKEN DETAILS")
print("=" * 80)
for i, token_id in enumerate(tokens):
try:
piece = llm.detokenize([token_id])
except Exception as exc:
piece = f"<ERROR: {exc}>".encode()
print(
f"{i:4d} "
f"id={token_id:8d} "
f"bytes={piece!r}"
)
print("\nSPACE-RUN TESTS")
print("=" * 80)
for spaces in [1, 2, 3, 4, 8, 16, 24, 32, 40, 44, 45, 48, 56, 58, 59, 63, 64, 65, 80, 96, 127, 128]:
data = (b" " * spaces) + b"X"
ids = llm.tokenize(data, add_bos=False)
out = llm.detokenize(ids)
ok = out == data
print(
f"{spaces:3d} spaces: "
f"{'OK ' if ok else 'FAIL'} "
f"tokens={ids}"
)
if not ok:
print(f" expected: {data!r}")
print(f" actual: {out!r}")
print(" individual token pieces:")
for token_id in ids:
piece = llm.detokenize([token_id])
print(f" {token_id:8d}: {piece!r}")
- etc.
I also built the bundled llama.cpp and ran the same test input through
llama-tokenize. Native llama.cpp produces the same token IDs as
llama-cpp-python, including the problematic long-whitespace tokens
(e.g. token 15270 for the 64-space run).
I have not pursued a native C++ detokenization reproducer further, but the
provided Python reproducer demonstrates that these tokens are returned by
tokenization and subsequently detokenized as empty byte strings.
Failure Logs
Prerequisites
Please answer the following questions for yourself before submitting an issue.
Expected Behavior
Tokenization followed by detokenization should reproduce the original byte sequence exactly.
In particular, runs of whitespace that are successfully represented by tokenizer token IDs should not be discarded during detokenization.
For example:
Python 3.12.3
GNU Make 4.3
g++ (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
$
I also built the bundled llama.cpp and ran the same test input through
llama-tokenize. Native llama.cpp produces the same token IDs asllama-cpp-python, including the problematic long-whitespace tokens
(e.g. token 15270 for the 64-space run).
I have not pursued a native C++ detokenization reproducer further, but the
provided Python reproducer demonstrates that these tokens are returned by
tokenization and subsequently detokenized as empty byte strings.
Failure Logs