Add RSVP continuous speech controller

[sajjad-mazaheri]

Summary

Adds the RSVP continuous speech controller for automatic speech responses.

What changed

• Adds an RSVP-specific automatic speech response mode using responseSpokenBool.
• Prepares the microphone and STT provider before stimulus presentation.
• Starts continuous audio capture at the first RSVP target onset.
• Allows provider finalization after the final target offset.
• Suppresses the matrix, keypad, and human-scoring controls only in automatic speech mode.
• Adds cleanup for trial skip, experiment quit, and page unload.
• Adds a simulation path that does not require a microphone or STT provider.

Testing

• TypeScript checks pass.
• RSVP speech mode, controller, runtime, and stimulus-generation tests pass.

[Copilot]

Pull request overview

Adds automatic speech recognition support to RSVP reading trials, including speech-mode handling, microphone/STT lifecycle management, simulation, and cleanup.

Changes:

• Adds RSVP speech runtime, mode, and controller logic.
• Integrates speech capture with RSVP timing and response handling.
• Adds lifecycle, simulation, and stimulus-generation tests.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Summary
threshold.js	Integrates automatic speech trials; critical completion, cleanup sequencing, and instruction issues remain.
tests/rsvpSpeechRuntime.test.ts	Tests speech runtime lifecycle and simulation.
tests/rsvpSpeechMode.test.ts	Tests response-mode resolution.
tests/rsvpSpeechController.test.ts	Tests speech controller lifecycle.
tests/onStimulusGeneratedRsvp.test.ts	Tests response-mode stimulus generation.
components/rsvpSpeech/rsvpSpeechRuntime.ts	Manages speech trial runtime state.
components/rsvpSpeech/rsvpSpeechMode.ts	Resolves RSVP response modes.
components/rsvpSpeech/rsvpSpeechController.ts	Controls microphone and provider lifecycle.
components/rsvpReading.js	Coordinates speech capture with RSVP timing; automatic completion remains unresolved.
components/onStimulusGenerated.ts	Supports mode-aware stimulus setup.
components/lifetime.js	Handles speech cleanup on quit and unload.
components/keypad.js	Restricts keypad handling by response mode.
components/global.js	Stores shared speech runtime state.

Suppressed comments (7)

components/rsvpReading.js:668

• allowProviderFinalization() only sets a flag; it does not request a provider commit. If VAD emits its commit while the targets are still on screen (for example, when the participant pauses), this call leaves that text buffered with no later commit, so the controller waits until its duration/finalization timeout. At the final target offset, force a provider commit or otherwise finalize the buffered transcript.

      typeof rsvpReadingTargetSets.current === "undefined" &&
      rsvpReadingTargetSets.upcoming.length === 0 &&
      isRsvpReadingAutomaticSpeechResponseMode(rsvpReadingResponse.responseType)
    )
      allowRsvpSpeechProviderFinalization();

components/rsvpReading.js:603

• Even if the automatic trial is made to exit, this path never persists its speech result. _rsvpReading_trialRoutineEnd records phraseIdentificationResponse, which automatic mode never fills, and getRsvpSpeechResult() has no production caller, so the transcript is lost from experiment data. Add an automatic-response data field and define its scoring/timing semantics before trial end.

    } else if (
      isRsvpReadingExperimenterResponseMode(responseMode) &&
      !rsvpReadingResponse.displayStatus
    ) {
      // Else create some subtle feedback that the scientist can use

components/rsvpSpeech/rsvpSpeechController.ts:135

• Optional chaining does not validate runtime types here: values such as utteranceId: 1, languageCode: {}, or targetWords: "cat" still call .trim()/.map() and throw a raw TypeError before the structured configuration checks run. Validate the input types before normalization so callers consistently receive RsvpSpeechControllerError with invalidConfiguration.

  const utteranceId = configuration.utteranceId?.trim();
  const languageCode = configuration.languageCode?.trim();
  const targetWords = configuration.targetWords?.map((word) => word.trim());

components/rsvpSpeech/rsvpSpeechRuntime.ts:318

• The completion handler stores the result only in rsvpSpeechRuntime; no production caller reads getRsvpSpeechResult, while _rsvpReading_trialRoutineEnd serializes phraseIdentificationResponse (which is never populated for automatic mode). Thus the captured transcript is not written to experiment data or converted into rsvpReadingResponsesBool/QUEST responses. Integrate the speech result into trial-end data/scoring before it is cleared.

        runtime.result = result;
        runtime.status = "completed";
        removePageHideListener();

threshold.js:6612

• prepareRsvpSpeechTrial catches configuration/provider failures and resolves false, but this await discards that result. The trial then proceeds with runtime.status === "failed"; the later capture/finalization calls return false, and because manual responses are disabled in automatic mode the trial cannot complete. Handle a false result here by aborting/skipping and reporting the failed trial before continuing.

      if (rsvpSpeechPreparationPromise) await rsvpSpeechPreparationPromise;

threshold.js:6211

• The simulation branch only puts the speech runtime into ready/capturing/finalizing states; it never calls injectRsvpSpeechTranscriptForSimulation. After the final target, the result remains undefined, while the simulated participant has no answer controls and automatic mode ignores its space key, so simulated automatic-speech trials cannot complete. Inject a deterministic transcript at the finalization point and route it through the same completion/scoring path.

              prepareSimulatedRsvpSpeechTrial({

threshold.js:7939

• This is the only new speech cleanup in the trial loop, but it is reached only when toShowCursor() is handled at the top of the routine. The continuous-gaze failure path calls skipTrial() and returns Scheduler.Event.NEXT earlier (threshold.js:8629–8633), bypassing this block and leaving the microphone/provider active through the skipped trial. Move cleanup to common trial-end/skip handling or add it to that early return.

            isRsvpReadingAutomaticSpeechResponseMode(
              rsvpReadingResponse.responseType,
            )
          )
            void closeActiveRsvpSpeechTrial();

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.