Fixing content parsing for onyx - #4448
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Onyx LLM streaming output parsing behavior to strip model framing tokens (e.g., to=user / <|message|>) from content deltas so clients don’t see those protocol markers in streamed responses.
Changes:
- Updated the Onyx streaming content-only unit test to include
<|start|>andassistantas separate initial chunks. - Adjusted Onyx tool parser “special tags to erase” to also erase
<|message|>, and changed the content-start indicator tokenization.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/test/llm/output_parsers/onyx_output_parser_test.cpp | Adjusts streaming test input chunks to match updated model framing emission. |
| src/llm/io_processing/onyx/onyx_tool_parser.hpp | Updates constants and the tag-erasure list used during streaming content parsing. |
| src/llm/io_processing/onyx/onyx_tool_parser.cpp | Updates the concrete tag strings for content framing removal. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
### 🛠 Summary JIRA/Issue if applicable. Describe the changes. ### 🧪 Checklist - [ ] Unit tests added. - [ ] The documentation updated. - [ ] Change follows security best practices. ``
dtrawins
approved these changes
Aug 14, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/test/llm/output_parsers/onyx_output_parser_test.cpp:677
- These added chunks do not cover the regression: the previous combined
to=user<|message|>tag already buffers and removes these adjacent chunks, so the test passes without the production change. Put the completed assistant prefix andto=userin the same chunk; under the old code, findingASSISTANT_PREFIXcaused the cache to flush the still-incompleteto=user<|message|>envelope and leakto=user, while the new separate tags prevent that.
{"<|start|>", ov::genai::GenerationFinishReason::NONE, std::nullopt},
{"assistant ", ov::genai::GenerationFinishReason::NONE, std::nullopt},
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠 Summary
Removing
to=user<|message|>from content with new version of model🧪 Checklist
``