Skip to content

test: synchronize bulkhead capacity tests on events instead of sleeps - #185

Merged
lesnik512 merged 1 commit into
mainfrom
test/bulkhead-fail-fast-sync
Oct 17, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
test/bulkhead-fail-fast-sync

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Closes #184

Problem

test_fail_fast_rejects_when_at_capacity in tests/test_bulkhead_props.py used time to synchronize. It started the holder requests, slept 5 ms on the assumption that they had all acquired their slots, and counted on _InFlightHandler(delay=0.05) keeping those slots until every extra request had been rejected. When the process is descheduled for longer than 50 ms, a holder's sleep timer fires during the extras loop, the holder releases its slot, and the next extra request goes through, so pytest.raises(BulkheadFullError) fails with DID NOT RAISE.

Several other bulkhead tests do the same thing: a fixed sleep or sleep(0) for "the holder has the slot", then a handler delay as the window in which the rejection must happen.

Reproduction (before)

Run on this branch's base (origin/main, 9025d02), on an 8-core Mac with 16 yes > /dev/null processes running (load average 27 to 43):

  • pytest tests/test_bulkhead_props.py::test_fail_fast_rejects_when_at_capacity --count=100 -x failed on repetition 7: FlakyFailure ... max_concurrent=1, extra_requests=7, Failed: DID NOT RAISE BulkheadFullError.
  • The four bulkhead test files at --count=30: the async props test failed 11 of 30 times and test_bulkhead_sync_props.py::test_fail_fast_rejects_when_at_capacity failed 1 of 30. That sync test already waits on a barrier for acquisition, but it still held the slots with time.sleep(0.05).

To show which other tests rely on the hold window, I ran a scratch copy of the four files with every "let the holder acquire" wait made longer than the handler's hold delay. That simulates a scheduler stall at that point. Ten tests failed:

  • test_bulkhead.py: test_bounded_wait_raises_bulkhead_full_error, test_acquire_timeout_zero_fails_fast, test_slot_released_on_cancellation, test_cancellation_before_acquire_does_not_hold_slot, test_bulkhead_full_error_is_not_retried_by_retry
  • test_bulkhead_props.py: test_fail_fast_rejects_when_at_capacity
  • test_bulkhead_sync.py: test_acquire_timeout_rejects_when_no_slot_available, test_bulkhead_full_error_no_chaining, test_emits_rejected_event
  • test_bulkhead_sync_props.py: test_fail_fast_rejects_when_at_capacity

test_raises_bulkhead_full_error_when_acquire_timeout_exceeded, test_bulkhead_full_error_chains_from_timeout and test_bulkhead_rejected_emits_observability_event in test_bulkhead.py only sleep(0) before the rejected call. They break only if the process stalls between the holder's first step and the test's next step for longer than the hold (1 s, 1 s and 50 ms), but they rely on the same window, so I moved them over too.

Fix

A _GatedHandler(capacity) in test_bulkhead.py, test_bulkhead_props.py and test_bulkhead_sync.py, an asyncio version in the first two and a threading version in the third. Each file already defines its own handlers, so I kept that pattern. The handler counts requests as they enter and sets full when capacity of them are inside. Those requests then wait on release, which the test sets after the rejections have been asserted. Requests beyond capacity return straight away, so if the bulkhead wrongly admits one, the test fails on pytest.raises and does not hang. The full wait has a 5 s timeout (asyncio.timeout or Event.wait(timeout=...)) that only guards against a hang. The tests no longer depend on timing to be correct.

In test_bulkhead_sync_props.py the existing _BarrierHandler now waits on a release event instead of sleeping 50 ms. Holders are released in a finally, so a failing example does not leave threads or tasks blocked.

The tests where a delay only creates overlap (test_serializes_at_capacity, test_max_concurrent_2_observes_at_most_2_in_flight, test_shared_bulkhead_enforces_joint_cap, test_acquire_timeout_none_waits_forever, the test_in_flight_never_exceeds_max_concurrent properties, test_no_slot_leak_after_drain, test_acquire_timeout_none_blocks_until_slot_available) pass whatever the scheduling, so I left them alone.

Verification (after)

Under the same 16-process load:

  • test_bulkhead_props.py::test_fail_fast_rejects_when_at_capacity --count=100: 100 passed (load average 15 to 51).
  • The four bulkhead test files at --count=30: 1260 passed. test_construct_outside_event_loop_then_use_inside was deselected for this run because it uses a module-scope AsyncBulkhead and fails on any repeat with the cross-loop guard, load or no load.

The scratch stall run, repeated on the fixed tests with a blocking 0.6 s sleep right after each full wait (0.06 s after the barrier in the sync props test): 43 passed.

just lint, just lint-ci and just test-ci (100% coverage) pass locally.

The fail-fast and rejection tests assumed a fixed sleep let the holders acquire their slots and that a handler delay kept them held until the rejections ran. Under CPU load the holders released early and the extra requests went through. The handlers now hold slots until the test releases them.
@lesnik512
lesnik512 merged commit b9e2861 into main Oct 17, 2026
13 checks passed
@lesnik512
lesnik512 deleted the test/bulkhead-fail-fast-sync branch October 17, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_fail_fast_rejects_when_at_capacity is timing-dependent and flaky under load

1 participant