fix(e2e): de-flake thread_resume by waiting on the resume replay - #3021
fix(e2e): de-flake thread_resume by waiting on the resume replay#3021dokterbob wants to merge 2 commits into
Conversation
`cypress/e2e/thread_resume/spec.cy.ts` failed ~20-33% of runs with a detached-DOM error when clicking `#chat-submit`, burning ~2 minutes of CI per occurrence and producing red builds that looked like real regressions on unrelated PRs. An assertion terminates a Cypress query chain: `.should()` freezes the element it resolved, so the following action command acts on that stale node instead of re-querying it. `SubmitButton` renders `#stop-button` and `#chat-submit` in different branches of a ternary, so the send button is genuinely unmounted while a task is running -- not merely disabled. On resume, `task_start`/`task_end` and `first_interaction` all fire while `@cl.on_chat_resume` runs, destroying and recreating `#chat-submit` in exactly the window between the assertion passing and the click landing. Split the chains in `submitMessage` and `openHistory` so Cypress keeps retrying the query itself. Helper signatures are unchanged, so all calling specs are fixed at once. Also wait for the thread to finish hydrating in the resume spec: `#message-composer` turns visible before resume completes, so assert the `on_chat_resume` message has landed and no task is running before submitting again. Note the test app's `MemoryDataLayer.create_step` is a no-op and threads store `steps: []`, so the first turn is not replayed on resume -- the `Resumed:` message is the signal that is actually there. Refs #3019 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cw4RvYzeQqLrEgvX2kehYK
The previous commit's post-reload waits were built on a misreading of the resume path; an adversarial review refuted both. `connection_successful` (socket.py:208-240) emits, in order: `first_interaction`, then the `@cl.on_chat_resume` message, then `resume_thread`. The client's `resume_thread` handler rebuilds the message list from `thread.steps` and *replaces* it wholesale. So the `Resumed: ...` message this spec waited for is erased milliseconds after it renders -- the assertion could only pass by sampling a sub-frame window. And there is no `task_start` on the resume path at all (only `task_end`, up front), so `#stop-button` never exists and asserting its absence guarded nothing. The real churn is composer ownership: `Footer` renders nothing while the message list is empty and `WelcomeScreen` renders its own `MessageComposer` instead, so every flip of `hasMessage()` remounts the input. With `create_step` a no-op and `steps` always `[]`, the replay drove the list back to empty and swapped the composer a second time -- landing squarely in the window the old assertion opened. Persist steps in the fixture's `MemoryDataLayer` so resume replays a real history. That gives the swap a stable end state (`hasMessage()` stays true across the replay) and makes the spec exercise thread resume rather than resuming an empty thread. Wait on `Echo: hi`, which only the replay can produce and which is the last write to the list. Also wait for the `on_chat_start` welcome message before the first submit: the same swap races `type()` on the first turn, dropping the typed value and leaving the send button disabled. The `testUtils` comment claimed `.should()` freezes its subject. The Cypress docs say no such thing -- they say queries are retried but action commands are not re-run, and recommend a separate `cy.get()` per action. Keep the split, correct the rationale. Refs #3019 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cw4RvYzeQqLrEgvX2kehYK
|
@codex review |
There was a problem hiding this comment.
1 issue found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="cypress/e2e/thread_resume/main.py">
<violation number="1" location="cypress/e2e/thread_resume/main.py:54">
P3: ensure_thread re-homes any thread whose stored userIdentifier differs from the user_id passed to update_thread, moving it out of whichever bucket currently holds it. In this shared multi-user store (alice and bob authenticate against the same global THREADS), an update_thread issued with a different owner_id would silently reassign another user's thread to the caller. The owner is the session user and is fixed per thread, so this is latent for the current test, but the guard offers no protection if a thread id is ever referenced from a second user's session. Consider skipping the re-home when the thread already has an owner, or guarding on ownership rather than unconditional reassignment.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| "steps": [], | ||
| } | ||
| THREADS.setdefault(user_id or "", []).append(thread) | ||
| elif user_id and thread.get("userIdentifier") != user_id: |
There was a problem hiding this comment.
P3: ensure_thread re-homes any thread whose stored userIdentifier differs from the user_id passed to update_thread, moving it out of whichever bucket currently holds it. In this shared multi-user store (alice and bob authenticate against the same global THREADS), an update_thread issued with a different owner_id would silently reassign another user's thread to the caller. The owner is the session user and is fixed per thread, so this is latent for the current test, but the guard offers no protection if a thread id is ever referenced from a second user's session. Consider skipping the re-home when the thread already has an owner, or guarding on ownership rather than unconditional reassignment.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At cypress/e2e/thread_resume/main.py, line 54:
<comment>ensure_thread re-homes any thread whose stored userIdentifier differs from the user_id passed to update_thread, moving it out of whichever bucket currently holds it. In this shared multi-user store (alice and bob authenticate against the same global THREADS), an update_thread issued with a different owner_id would silently reassign another user's thread to the caller. The owner is the session user and is fixed per thread, so this is latent for the current test, but the guard offers no protection if a thread id is ever referenced from a second user's session. Consider skipping the re-home when the thread already has an owner, or guarding on ownership rather than unconditional reassignment.</comment>
<file context>
@@ -23,6 +24,42 @@
+ "steps": [],
+ }
+ THREADS.setdefault(user_id or "", []).append(thread)
+ elif user_id and thread.get("userIdentifier") != user_id:
+ for uid, threads in THREADS.items():
+ THREADS[uid] = [t for t in threads if t is not thread]
</file context>
Fixes #3019.
What the flake actually is
The issue diagnosed a stale-subject bug in
submitMessage—cy.get('#chat-submit').should('not.be.disabled').click()acting on an already-resolved node. That's a real smell and is fixed here, but it isn't the mechanism.Composer ownership moves between two different components as the message list changes:
frontend/src/components/chat/Footer.tsx:20returnsnullwhile!hasMessage(messages)frontend/src/components/chat/WelcomeScreen.tsx:80,91returnsnulloncehasMessage(messages), and otherwise renders its ownMessageComposerSo every flip of
hasMessage()unmounts one composer and mounts another. Two points in this spec crossed that boundary mid-interaction:on_chat_startsends its welcome message asynchronously. Landing duringtype()remounts the input, loses the typed value, and leaves#chat-submitdisabled.connection_successful(backend/chainlit/socket.py:208-240) emitsfirst_interaction→ the@cl.on_chat_resumemessage →resume_thread. The client handler (libs/react-client/src/useChatSession.ts:258+) rebuilds the list fromthread.stepsand replaces it wholesale. Because this fixture'sMemoryDataLayer.create_stepwas a no-op andstepswas always[], that replay drove the list back to empty — swapping the composer a second time, exactly where the test clicks.cy.get('#message-composer').should('be.visible')is satisfied early in that sequence, well before hydration finishes, so it never gated anything.The underlying app behaviour is filed separately as #3022 — this PR makes the test honest about the churn, it does not remove the churn.
Changes
cypress/e2e/thread_resume/main.py— implementcreate_step/update_step/delete_stepon the fixture'sMemoryDataLayerso steps are actually persisted, with@queue_until_user_message()matchingdata/sql_alchemy.py:383-421.ensure_threadbackfills a row when a step is persisted before the firstupdate_thread, and re-homes it when the owner arrives rather than duplicating it.This is the crux, not incidental cleanup. With
steps: []the resume ends in a state indistinguishable from a fresh chat, so there was no observable signal that hydration had finished to wait on. Persisting steps gives the replay a stable end state —hasMessage()now stays true across it, so the composer no longer remounts — and means the spec finally exercises thread resume instead of resuming an empty thread.cypress/e2e/thread_resume/spec.cy.ts— wait for the welcome message before the first submit, and after reload wait onEcho: hi, which only the replay can produce and which is the last write to the message list. Nocy.wait(<ms>).cypress/support/testUtils.ts— split the query chains insubmitMessage/openHistoryso each action re-queries. Signatures unchanged; all 34 call sites unaffected.Verification
Local:
pnpm lint cypress/, Prettier, andruff check/ruff format --checkall clean. Notepnpm type-checkdoes not covercypress/.CI: three attempts of run 32954779496, all on this commit, all green. Because
retries: 3would mask a surviving flake behind a green badge, I pulled all 30 shard logs and checked(Attempt N of 4)annotations directly:thread_resume(ubuntu-5)thread_resume(windows-5)Six first-attempt passes, no retry annotations on this spec anywhere. All 30 shard-runs ended
Failing: 0.Being straight about the strength of this: at the 20–33% rate reported in #3019, three clean runs would happen by luck roughly 30–50% of the time, so the runs alone aren't proof. The mechanism argument is the stronger half —
hasMessage()no longer flips during the resume, so the remount that detached the button cannot occur — and the retry data rules out the specific way a survivor would have hidden.Attempt 3 did contain one genuine retry, in
oauth_authonwindows-latest-3, unrelated to this spec and pre-existing. Filed as #3023, with the general "retries are invisible in CI" problem as #3024.Review notes
2099804is wrong andc13ae25corrects it; left in history rather than force-pushed so the reasoning is visible. Happy to squash on merge.submitMessageis still exposed to the same remount race in other specs that call it against an empty message list (Composer remounts whenever the message list transitions empty↔non-empty, dropping in-progress input #3022 is the real fix);[data-testid="read-only-banner"]atspec.cy.ts:48asserts a testid that exists nowhere in the frontend and is therefore vacuous (noted in Surface Cypress retries in CI so flaky specs stop hiding behind green runs #3024); ~27 other.should(...).<action>()sites remain acrosscypress/e2e/.