Repository navigation
test: synchronize bulkhead capacity tests on events instead of sleeps - #185
Merged
Merged
Conversation
The fail-fast and rejection tests assumed a fixed sleep let the holders acquire their slots and that a handler delay kept them held until the rejections ran. Under CPU load the holders released early and the extra requests went through. The handlers now hold slots until the test releases them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #184
Problem
test_fail_fast_rejects_when_at_capacityintests/test_bulkhead_props.pyused time to synchronize. It started the holder requests, slept 5 ms on the assumption that they had all acquired their slots, and counted on_InFlightHandler(delay=0.05)keeping those slots until every extra request had been rejected. When the process is descheduled for longer than 50 ms, a holder's sleep timer fires during the extras loop, the holder releases its slot, and the next extra request goes through, sopytest.raises(BulkheadFullError)fails withDID NOT RAISE.Several other bulkhead tests do the same thing: a fixed sleep or
sleep(0)for "the holder has the slot", then a handler delay as the window in which the rejection must happen.Reproduction (before)
Run on this branch's base (
origin/main, 9025d02), on an 8-core Mac with 16yes > /dev/nullprocesses running (load average 27 to 43):pytest tests/test_bulkhead_props.py::test_fail_fast_rejects_when_at_capacity --count=100 -xfailed on repetition 7:FlakyFailure ... max_concurrent=1, extra_requests=7,Failed: DID NOT RAISE BulkheadFullError.--count=30: the async props test failed 11 of 30 times andtest_bulkhead_sync_props.py::test_fail_fast_rejects_when_at_capacityfailed 1 of 30. That sync test already waits on a barrier for acquisition, but it still held the slots withtime.sleep(0.05).To show which other tests rely on the hold window, I ran a scratch copy of the four files with every "let the holder acquire" wait made longer than the handler's hold delay. That simulates a scheduler stall at that point. Ten tests failed:
test_bulkhead.py:test_bounded_wait_raises_bulkhead_full_error,test_acquire_timeout_zero_fails_fast,test_slot_released_on_cancellation,test_cancellation_before_acquire_does_not_hold_slot,test_bulkhead_full_error_is_not_retried_by_retrytest_bulkhead_props.py:test_fail_fast_rejects_when_at_capacitytest_bulkhead_sync.py:test_acquire_timeout_rejects_when_no_slot_available,test_bulkhead_full_error_no_chaining,test_emits_rejected_eventtest_bulkhead_sync_props.py:test_fail_fast_rejects_when_at_capacitytest_raises_bulkhead_full_error_when_acquire_timeout_exceeded,test_bulkhead_full_error_chains_from_timeoutandtest_bulkhead_rejected_emits_observability_eventintest_bulkhead.pyonlysleep(0)before the rejected call. They break only if the process stalls between the holder's first step and the test's next step for longer than the hold (1 s, 1 s and 50 ms), but they rely on the same window, so I moved them over too.Fix
A
_GatedHandler(capacity)intest_bulkhead.py,test_bulkhead_props.pyandtest_bulkhead_sync.py, an asyncio version in the first two and a threading version in the third. Each file already defines its own handlers, so I kept that pattern. The handler counts requests as they enter and setsfullwhencapacityof them are inside. Those requests then wait onrelease, which the test sets after the rejections have been asserted. Requests beyondcapacityreturn straight away, so if the bulkhead wrongly admits one, the test fails onpytest.raisesand does not hang. Thefullwait has a 5 s timeout (asyncio.timeoutorEvent.wait(timeout=...)) that only guards against a hang. The tests no longer depend on timing to be correct.In
test_bulkhead_sync_props.pythe existing_BarrierHandlernow waits on areleaseevent instead of sleeping 50 ms. Holders are released in afinally, so a failing example does not leave threads or tasks blocked.The tests where a delay only creates overlap (
test_serializes_at_capacity,test_max_concurrent_2_observes_at_most_2_in_flight,test_shared_bulkhead_enforces_joint_cap,test_acquire_timeout_none_waits_forever, thetest_in_flight_never_exceeds_max_concurrentproperties,test_no_slot_leak_after_drain,test_acquire_timeout_none_blocks_until_slot_available) pass whatever the scheduling, so I left them alone.Verification (after)
Under the same 16-process load:
test_bulkhead_props.py::test_fail_fast_rejects_when_at_capacity --count=100: 100 passed (load average 15 to 51).--count=30: 1260 passed.test_construct_outside_event_loop_then_use_insidewas deselected for this run because it uses a module-scopeAsyncBulkheadand fails on any repeat with the cross-loop guard, load or no load.The scratch stall run, repeated on the fixed tests with a blocking 0.6 s sleep right after each
fullwait (0.06 s after the barrier in the sync props test): 43 passed.just lint,just lint-ciandjust test-ci(100% coverage) pass locally.