Skip to content

examples: split vLLM generation across prefill and decode sessions - #515

Open
k82cn wants to merge 1 commit into
xflops:mainfrom
k82cn:examples/vllm-prefill-decode
Open

examples: split vLLM generation across prefill and decode sessions#515
k82cn wants to merge 1 commit into
xflops:mainfrom
k82cn:examples/vllm-prefill-decode

Conversation

@k82cn

@k82cn k82cn commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Turn the vLLM smoke test into a Runner example with two sessions: PrefillEngine and DecodeEngine.
  • Prefill returns a PrefillOutput object; decode continues from that object via TokensPrompt.
  • This is a session-level split (decode still recomputes KV), not vLLM KV-connector disaggregation.

Test plan

  • From the Flame console: cd /opt/examples/vllm && uv run main.py
  • Confirm two sessions (vllm-prefill, vllm-decode) and that generated text is printed
  • Optional: set VLLM_MODEL and re-run with another small model

Use Runner services so a prompt is prefilled on one session and decoded on another.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant