Skip to content

fix(audio): keep the selected command on voice/STT-dictated messages - #2992

Open
RoyBA wants to merge 8 commits into
Chainlit:mainfrom
RoyBA:fix/stt-command-auto-attach
Open

fix(audio): keep the selected command on voice/STT-dictated messages#2992
RoyBA wants to merge 8 commits into
Chainlit:mainfrom
RoyBA:fix/stt-command-auto-attach

Conversation

@RoyBA

@RoyBA RoyBA commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Problem

When a command is selected in the composer (e.g. /search) and the user dictates their message with the voice/STT feature instead of typing it, the message is sent without the selected command. Typing the exact same message correctly includes the command.

Root cause

The selected command lives in frontend state (persistentCommandState) and is attached to the message only in the text-composer submit path (the client_message event). The audio path (audio_startaudio_chunk
audio_end) never transmits the command, and the transcribed user message is created on the backend (in the app's on_audio_end handler), so it has no knowledge of the UI selection.

Fix

Carry the selected command through the audio turn and auto-attach it to the user message produced during that turn.

Frontend — send the selected command with audio_start:

  • VoiceButton reads persistentCommandState and passes selectedCommand?.id to startConversation.
  • useAudio.startConversation(command?)useChatInteract.startAudioStream(command?)socket.emit('audio_start', { command }).

Backend — store it for the turn and attach it:

  • BaseSession gains a current_command field.
  • audio_start stores session.current_command; audio_end clears it, so it only applies to that audio turn.
  • Message.__post_init__ auto-attaches current_command to user_messages that don't already carry a command.

Backward compatibility

Fully backward compatible. The audio_start payload is optional (old clients that emit no payload keep working), the new frontend arguments are optional, and the existing Message.command field is reused. Typed messages and assistant messages are unchanged.

Testing

  • Added unit tests in backend/tests/test_message.py: auto-attach on user_message, explicit command takes precedence, no-op when no command is set, and assistant messages never inherit the command.
  • Full backend suite, mypy, ruff, pnpm type-check, ESLint, and Prettier all pass.
  • Manual check: select a command, dictate a message via the mic → the resulting user message carries the command, identical to typing it.

Out of scope

modes has the same gap (also only attached in the text submit path). Left for a follow-up to keep this PR focused on commands.


Summary by cubic

Voice/STT-dictated messages now keep the selected command like typed messages; previously they dropped it.

  • Frontend sends the selected command id with audio_start; the payload is optional, so old clients keep working.
  • Backend stores the command on the session only when the audio start is accepted, and clears it on audio_end.
  • Message.__post_init__ auto-attaches the session command to command-less user_messages; from_dict resets from the payload so typed or resumed messages never inherit an active audio turn's command.

Written for commit 93d7102. Summary will update on new commits.

Review in cubic

Auto-attach the UI-selected command to the user message produced during an audio turn, so voice-dictated messages carry the command just like typed ones. The command is sent with audio_start, stored on the session for the turn, cleared on audio_end, and attached in Message.__post_init__.

Co-Authored-By: GitHub Copilot <noreply@github.com>
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. backend Pertains to the Python backend. bug Something isn't working frontend Pertains to the frontend. unit-tests Has unit tests. labels Jul 28, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread backend/chainlit/message.py
Message.from_dict deserializes incoming client messages and resumed thread history. Because the audio-command fallback lives in the shared __post_init__, a command-less typed message (or resumed step) created during an active audio turn would wrongly inherit the turn's command. Reset the command from the payload in from_dict so deserialized messages stay authoritative; app-constructed transcription messages still inherit as intended.

Addresses PR review feedback.

Co-Authored-By: GitHub Copilot <noreply@github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Carries selected commands through voice/STT turns so dictated messages match typed-message behavior.

Changes:

  • Sends the selected command with audio_start.
  • Stores and attaches the command to generated user messages.
  • Adds command inheritance and deserialization tests.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
libs/react-client/src/useChatInteract.ts Emits command with audio start.
libs/react-client/src/useAudio.ts Forwards command when starting audio.
frontend/src/components/chat/MessageComposer/VoiceButton.tsx Reads and passes the selected command.
backend/chainlit/socket.py Manages command state across audio turns.
backend/chainlit/session.py Adds session command state.
backend/chainlit/message.py Attaches commands to generated user messages.
backend/tests/test_message.py Tests command attachment behavior.
Suppressed comments (1)

backend/chainlit/socket.py:492

  • This unconditional clear races with a subsequent audio turn. endConversation marks the client off before emitting audio_end, while this handler awaits the app's on_audio_end; a quick new audio_start can therefore overwrite current_command, after which the older handler both exposes the new command to the old callback and clears it here. Associate the command with a turn/context (or serialize audio turns) so overlapping handlers cannot misattribute or erase commands.
        session.current_command = None

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread backend/chainlit/socket.py Outdated
dokterbob and others added 5 commits August 17, 2026 18:14
A declined on_audio_start (or disabled audio) previously left the UI-selected command on the session. Since the frontend sends no audio_end for a rejected start, the turn cleanup never ran and a later server-created user message could inherit the stale command. Record the command only once the connection is accepted.

Addresses PR review feedback.

Co-Authored-By: GitHub Copilot <noreply@github.com>
@dokterbob

Copy link
Copy Markdown
Collaborator

@codex review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Pertains to the Python backend. bug Something isn't working frontend Pertains to the frontend. size:M This PR changes 30-99 lines, ignoring generated files. unit-tests Has unit tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants