Feature/rsvp speech phase5 - #138
Merged
Merged
Conversation
- Introduced ignoreWordOrder parameter to allow flexible scoring in RSVP speech responses. - Updated scoring policies to support both strict and any order matching. - Modified response registration logic to ensure automatic speech responses are handled correctly. - Added comprehensive test cases for various speech scoring scenarios, including edge cases for word order and repetition. - Refactored related components to improve clarity and maintainability, ensuring backward compatibility with existing functionality. - Introduced a new JSON fixture for golden test cases to validate scoring behavior against expected outcomes.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The automatic-speech validity logic can misclassify pre-exposure failures as post-exposure when registration is marked invalid, causing unseen target words to be incorrectly consumed.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR advances the RSVP “automaticSpeech” pathway by separating response timing vs STT finalization timing, adding an explicit response-registration step/state, and strengthening scoring/validity rules so QUEST only consumes scientifically valid automatic-speech trials.
Changes:
- Introduces automatic-speech response registration (state + registrar) and threads registration status into QUEST validity decisions.
- Splits RSVP speech timing into “beyond final word” response window vs STT finalization timeout; updates glossary parameter names accordingly.
- Expands and hardens scoring/validity test coverage, including a golden-fixture suite for scoring behavior.
File summaries
| File | Description |
|---|---|
| threshold.js | Updates RSVP speech trial setup fields and adds finalization timeout parameter wiring. |
| components/trialRoutines.js | Passes automatic-speech registration status into validity checks used to decide QUEST consumption/retry. |
| components/rsvpSpeech/rsvpSpeechValidity.ts | Tightens validity gating for automatic-speech trials using registration status. |
| components/rsvpSpeech/rsvpSpeechScoring.ts | Ensures only approved exact-match kinds can score as correct (no fuzzy correctness). |
| components/rsvpSpeech/rsvpSpeechRuntime.ts | Updates glossary parameter names; splits response vs finalization timeouts; adds registration state to runtime resets. |
| components/rsvpSpeech/rsvpSpeechRegistrar.ts | Adds registrar to score and populate legacy response registers from STT results with explicit registration status. |
| components/rsvpReading.js | Integrates registrar into the RSVP response phase; supports simulation transcript injection; adjusts trial completion/counter behavior. |
| components/onStimulusGenerated.ts | Resets automatic-speech registration status per trial (only in automatic-speech mode). |
| components/global.js | Extends global RSVP speech runtime with registration status/error/diagnostics fields. |
| tests/rsvpSpeechValidity.test.ts | Adds coverage for pending/invalid registration states affecting validity. |
| tests/rsvpSpeechScoringGolden.test.ts | Adds golden-case test runner for scoring fixture cases. |
| tests/rsvpSpeechScoring.test.ts | Adds regression test ensuring near-matches never become correct in strict/anyOrder modes. |
| tests/rsvpSpeechRuntime.test.ts | Updates configuration tests for new parameter names and separate response vs finalization waits. |
| tests/rsvpSpeechRegistrar.test.ts | Adds unit tests for registrar behavior (registered/invalid/waiting and policy mapping). |
| tests/rsvpSpeechLegacyIsolation.sourceContracts.test.ts | Adds “source contract” checks to prevent leaking new speech-only outputs into legacy paths. |
| tests/onStimulusGeneratedRsvp.test.ts | Verifies per-trial reset does not affect legacy modes and only applies to automatic-speech. |
| tests/fixtures/rsvpSpeechScoringCases.json | Adds proposed human-readable scoring cases fixture used by golden tests. |
Review details
- Files reviewed: 17/17 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+39
to
+43
| if (input.responseRegistrationStatus === "invalid") { | ||
| return { | ||
| validForQuest: false, | ||
| consumeTargetWords: true, | ||
| invalidReason: "postExposureTechnicalFailure", |
| }): RsvpSpeechScoringPolicy => { | ||
| if (input.policy) return input.policy; | ||
| if (typeof input.ignoreWordOrder !== "boolean") { | ||
| throw new Error("rsvpReadingSTTIgnoreOrderBool must be a Boolean value."); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.