feat(gateway): wire VLM image/video/audio understanding via adapter - #3668
Open
jeffwu-1999 wants to merge 2 commits into
Open
feat(gateway): wire VLM image/video/audio understanding via adapter#3668jeffwu-1999 wants to merge 2 commits into
jeffwu-1999 wants to merge 2 commits into
Conversation
Base branch for the incremental gateway split. Adds the gateway SDK core (self-contained, stdlib-only) and an empty modality/__init__ so importing nexent.core.gateway works but the adapter registry is empty. - sdk/nexent/core/gateway/: __init__ (imports modality for side-effect registration), registry, transport, multimodal_adapter, multimodal_gateway, model_context (all ModelContext subclasses) - modality/__init__.py: empty aggregator (no per-vendor imports yet; each feature branch appends its modality's lines) - backend/services/model_gateway_service.py: bridge skeleton — generic machinery only (_FACTORY_NORMALIZE, _normalize_factory, _coalesce, _config_to_context with common-kwargs+raise, get_adapter_from_config, build_adapter_fresh, _fetch_slot_config, _fetch_voice_config). No per-modality wrappers/branches; nothing on develop imports this file yet. - sdk/pyproject.toml: namespaces=true (modality subpackages are PEP 420) Verified: py_compile OK; registry empty (has(openai,vlm)=False) as expected. Co-Authored-By: Claude <noreply@anthropic.com>
jeffwu-1999
changed the title
jeffwu-1999
force-pushed
the
feat/gw-vlm
branch
2 times, most recently
from
August 14, 2026 07:47
b377bb6 to
0fc22c4
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
jeffwu-1999
force-pushed
the
feat/gw-vlm
branch
from
August 14, 2026 08:16
0fc22c4 to
3a7bd09
Compare
First feature branch on top of the B0 core. Switches VLM model access from
image_service.get_vlm_model / get_video_understanding_model to the gateway
adapter (get_vlm_adapter / OpenAIVLMAdapter). LLM access is untouched here
(still get_llm_model — lands in B2).
- modality/vlm/{vlm_adapter,openai,modelengine}.py + 3 vlm import lines in
modality/__init__ (registers openai/modelengine vlm adapters)
- bridge: _config_to_context vlm branch + get_vlm_adapter /
get_vlm_adapter_from_config wrappers
- create_agent_info: AnalyzeImageTool (slot="vlm") / AnalyzeAudio|VideoTool
(slot="vlm3") injection switched to get_vlm_adapter
- tool_configuration_service: analyze_image/analyze_audio|video validation
switched to get_vlm_adapter
- model_health_service: vlm connectivity check via build_adapter_fresh.health_check
- SDK tools analyze_image/video/audio (VLMRequest + invoke_sync) +
test_vlm_adapter + test_analyze_image_tool
- tests: switched only the VLM patch sites in test_create_agent_info /
test_tool_configuration to get_vlm_adapter (slot="vlm"/"vlm3") + added the
model_gateway_service stub; LLM patches left as get_llm_model (B2)
Verified: py_compile OK; working-tree vlm slice == ac91318 final (diff vs
final shows only llm differences). E2E image-understanding path matches the
previously-verified source branch (analyze_image -> get_vlm_adapter ->
OpenAIVLMAdapter -> SiliconFlow).
Co-Authored-By: Claude <noreply@anthropic.com>
jeffwu-1999
force-pushed
the
feat/gw-vlm
branch
from
August 14, 2026 08:52
3a7bd09 to
021e935
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Move VLM model access (image / video / audio understanding) off image_service.get_vlm_model / get_video_understanding_model and onto the unified gateway adapter layer (get_vlm_adapter / OpenAIVLMAdapter).
Add the VLM modality package under gateway/modality/ (vlm_adapter, openai, modelengine) and register the adapters in modality/init.py
Extend the backend bridge: VLM branch in _config_to_context, plus get_vlm_adapter / get_vlm_adapter_from_config wrappers
Switch all VLM injection points to the adapter:create_agent_info: AnalyzeImageTool (slot vlm), AnalyzeAudioTool / AnalyzeVideoTool (slot vlm3)
tool_configuration_service: analyze_image / analyze_audio|video validation
model_health_service: VLM connectivity checks now go through build_adapter_fresh(...).health_check()
Rework SDK tools (analyze_image_tool.py, analyze_audio_tool.py, analyze_video_tool.py) to use VLMRequest + invoke_sync
Cover the new surface with tests: test_vlm_adapter.py, the full test/sdk/core/gateway/ suite, and test_model_gateway_service.py
Why
This is the second branch of a six-part gateway refactor (base: feat/gw-core). VLM image understanding is the already-validated capability, so it lands first as an independently reviewable and mergeable increment. LLM access intentionally stays on the legacy get_llm_model here so the VLM path can be reviewed in isolation before B2 wires the LLM.
Validation
py_compile passes on all touched SDK and backend files
All VLM-related files are byte-identical to the previously verified reference tree (ac91318); the only remaining diff is the expected LLM portion
E2E image-understanding path confirmed: analyze_image → get_vlm_adapter → OpenAIVLMAdapter → SiliconFlow 200
Coverage: patch gap closed from 113 uncovered lines to 100% line + branch coverage on all affected files (model_gateway_service.py + the full sdk/nexent/core/gateway/ package); local run: 83 passed, ruff clean
Frontend untouched in this branch
image
video
