stream: fix flaky stream destroy, reachable from HTTP/2 teardown - #65079
stream: fix flaky stream destroy, reachable from HTTP/2 teardown#65079pimterry wants to merge 1 commit into
Conversation
Signed-off-by: Tim Perry <pimterry@gmail.com>
|
cc @nodejs/http2 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #65079 +/- ##
==========================================
+ Coverage 90.29% 90.31% +0.01%
==========================================
Files 759 759
Lines 247598 247756 +158
Branches 46680 46720 +40
==========================================
+ Hits 223566 223749 +183
+ Misses 15503 15478 -25
Partials 8529 8529
🚀 New features to boost your workflow:
|
Stress testing inconclusive - the tests passed on both main and this branch, even with multiple runs. They're definitely failing in PRs though. I can reliably reproduce the reported failure locally (that first failure above, it hits for 2-3% of runs) and validate that it's fixed with this change. The trace is where is crashes on my machine with the matching error, and it's clearly a broken flow, so I think this is the right fix regardless. |
mcollina
left a comment
There was a problem hiding this comment.
lgtm
I concur with the analysis
Some of the HTTP/2 tests have become flaky, e.g. nodejs/reliability#1623 shows yesterday:
Best guess is these are both due to the window update PR #64623 (cc @mcollina) since the timing lines up exactly. I think this is really just the window size highlighting existing issues though.
This PR fixes the first issue, which triggers flakes in parallel/test-worker-terminate-http2-respond-with-file. I'll kick off a stress test to confirm, but I can reproduce this locally, and reproduced as resolved with this fix. I'm still working on the 2nd, which only reproduces on Mac and seems a bit more complex.
Actual failure in the 1st test is a crash with
pure virtual method called. That fires due toReadStopwithin this trace:I.e. during destroy within the destructor chain, we call OnStreamRead, which tries to call
stream()->ReadStopinside the sources destructor.This is only called because OnStreamDestroy manually calls OnStreamRead with an error code to reuse its error/eof teardown logic. We don't need most of that in the destruction scenario (which is only ever called from
~StreamResource, where the stream is already dead).I've refactored out the relevant bit to split them up (just guarding just fails in the next line, where
previous_listener_is also null). That then exposed two other bugs for the same state, where two other methods that get reached later in this teardown flow also fail to check if the stream is already destroyed - those just need simple guards.Seems like this is a flaky race because it depends on whether the sink (Http2Stream) or the file handle gets destroyed first by
Cleanup().