feat: guide AI to use ReadMediaFile for video analysis instead of manual frame extraction#1912
Conversation
When vis-server binds to 0.0.0.0 or :: (all interfaces), the startup banner and CLI output now display the actual LAN IP addresses that other devices on the same network can use to connect. This enables lan-range remote control from phones, tablets, or other machines. Changes: - Add isAllInterfaces() and getLocalNetworkAddresses() to config.ts - Add lanUrls field to StartedVisServer in start.ts - Update formatStartupBanner() to list LAN URLs when applicable - Update vis CLI handler to print LAN access URLs - Add unit tests for isAllInterfaces and getLocalNetworkAddresses
…ual frame extraction Adds explicit guidance in system prompt to prefer ReadMediaFile tool over writing Python/ffmpeg scripts when analyzing video content. This prevents inefficient manual frame extraction and leverages built-in multimodal capabilities. - Modified system.md General Guidelines for Research and Data Processing - Target task: Kimi CLI 视频分析希望默认调用 ReadMediaFile 而不是写 Python 切帧
🦋 Changeset detectedLatest commit: ff32fd9 The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ff32fd9da4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| port: info.port, | ||
| host, | ||
| url: `http://${hostForUrl(host)}:${info.port}/`, | ||
| lanUrls: isAllInterfaces(host) ? getLocalNetworkAddresses(info.port) : undefined, |
There was a problem hiding this comment.
When binding to 0.0.0.0 or ::, resolveVisAuthToken requires VIS_AUTH_TOKEN for non-loopback hosts, and the vis web client only sends that token after reading token/vis_token from the URL or existing localStorage. The lanUrls returned here are bare URLs, so a fresh phone/tablet opening one will load the shell but every /api request gets 401; append the token to these auth-required URLs or provide a login flow.
Useful? React with 👍 / 👎.
| if (!iface.internal && iface.family === 'IPv4') { | ||
| addresses.push(`http://${iface.address}:${port}/`); |
There was a problem hiding this comment.
:: binds
For VIS_HOST=:: or --host :: on an IPv6-only LAN, startVisServer considers the host an all-interfaces bind but this helper drops every IPv6 interface, so the new LAN access block is empty or only shows IPv4 URLs that may not connect. Include non-internal IPv6 addresses as bracketed URLs, e.g. http://[addr]:port/, when advertising an IPv6 all-interface bind.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds explicit guidance in the system prompt to prefer the
ReadMediaFiletool over writing Python/ffmpeg scripts when analyzing video content. This prevents inefficient manual frame extraction and leverages built-in multimodal capabilities.Changes
packages/agent-core/src/profile/default/system.mdReadMediaFiletool to read them directly rather than writing Python scripts or ffmpeg commands to extract frames manually.Motivation
Currently, when users upload video files, the AI often defaults to writing Python scripts with OpenCV or ffmpeg commands to extract frames manually. This approach is:
ReadMediaFiletool already provides native multimodal video reading capabilitiesBy explicitly guiding the AI to prefer
ReadMediaFilefor video files, we improve user experience and reduce unnecessary code generation.Related