Skip to content

Fixing content parsing for onyx - #4448

Merged
dtrawins merged 2 commits into
mainfrom
przepeck/onyx_content_fixes
Aug 14, 2026
Merged

Fixing content parsing for onyx#4448
dtrawins merged 2 commits into
mainfrom
przepeck/onyx_content_fixes

Conversation

@przepeck

Copy link
Copy Markdown
Collaborator

🛠 Summary

Removing to=user<|message|> from content with new version of model

🧪 Checklist

  • Unit tests added.
  • The documentation updated.
  • Change follows security best practices.
    ``

Copilot AI lite review requested due to automatic review settings August 13, 2026 14:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Onyx LLM streaming output parsing behavior to strip model framing tokens (e.g., to=user / <|message|>) from content deltas so clients don’t see those protocol markers in streamed responses.

Changes:

  • Updated the Onyx streaming content-only unit test to include <|start|> and assistant as separate initial chunks.
  • Adjusted Onyx tool parser “special tags to erase” to also erase <|message|>, and changed the content-start indicator tokenization.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
src/test/llm/output_parsers/onyx_output_parser_test.cpp Adjusts streaming test input chunks to match updated model framing emission.
src/llm/io_processing/onyx/onyx_tool_parser.hpp Updates constants and the tag-erasure list used during streaming content parsing.
src/llm/io_processing/onyx/onyx_tool_parser.cpp Updates the concrete tag strings for content framing removal.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/llm/io_processing/onyx/onyx_tool_parser.cpp
Comment thread src/llm/io_processing/onyx/onyx_tool_parser.hpp
Comment thread src/test/llm/output_parsers/onyx_output_parser_test.cpp
### 🛠 Summary

JIRA/Issue if applicable.
Describe the changes.

### 🧪 Checklist

- [ ] Unit tests added.
- [ ] The documentation updated.
- [ ] Change follows security best practices.
``

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/test/llm/output_parsers/onyx_output_parser_test.cpp:677

  • These added chunks do not cover the regression: the previous combined to=user<|message|> tag already buffers and removes these adjacent chunks, so the test passes without the production change. Put the completed assistant prefix and to=user in the same chunk; under the old code, finding ASSISTANT_PREFIX caused the cache to flush the still-incomplete to=user<|message|> envelope and leak to=user, while the new separate tags prevent that.
        {"<|start|>", ov::genai::GenerationFinishReason::NONE, std::nullopt},
        {"assistant ", ov::genai::GenerationFinishReason::NONE, std::nullopt},

@dtrawins
dtrawins merged commit 9bbe857 into main Aug 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants