feat: circuit breaker half-open state with rate-limited probe - #1373
Open
AgilityB wants to merge 1 commit into
Open
Conversation
…aceful degradation integration Enhance the circuit breaker's half-open state with a comprehensive rate-limited probe mechanism that integrates cooldown enforcement, quota-based admission, and explicit probe tracking, with graceful degradation oracle health integration. ### Changes **circuit_breaker.rs:** - `probe_request()` - Primary entry-point combining cooldown + quota-based admission - `half_open_probe_success()` - Records successful probe, auto-closes at threshold - `half_open_probe_failure()` - Records failed probe, re-opens breaker immediately - `reset_half_open_window()` - Cleans up temporary window data on state transitions **graceful_degradation.rs:** - `probe_oracle_with_circuit_breaker()` - Integrates CB half-open probe with oracle health checks **circuit_breaker_tests.rs:** - 9 new tests covering: quota admission, quota exhaustion, cooldown enforcement, success threshold closing, failure re-opening, window reset, quota+failures re-open, and full oracle integration
|
@AgilityB Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the circuit breaker HalfOpen state with a rate-limited probe mechanism for the Predictify Hybrid smart contract.
This PR adds a complete three-state circuit breaker (Closed → Open → HalfOpen → Closed/Open) with a quota-based probe scheduler that prevents flapping and provides controlled recovery testing.
Key Features
Cooldown enforcement: probes are not counted until recovery_timeout seconds have elapsed since entering HalfOpen
Quota-based scheduling via HalfOpenWindow with configurable calls_per_minute and evaluation_window_s
Returns Ok(true) when probe is admitted, Ok(false) when quota is full or cooldown active
2. Explicit Probe Tracking (half_open_probe_success / half_open_probe_failure)
half_open_probe_success: increments HalfOpenWindow.completed, delegates to record_success() which auto-closes after half_open_max_requests consecutive successes
half_open_probe_failure: increments HalfOpenWindow.failures, delegates to record_failure() which re-opens the breaker immediately
reset_half_open_window: cleans up temporary window data on state transitions
3. Graceful Degradation Integration (probe_oracle_with_circuit_breaker)
Full integration between circuit breaker half-open probe and oracle health checks
Flow: probe_request() → oracle health check → half_open_probe_success() / half_open_probe_failure()
Returns OracleHealth::Working, OracleHealth::Degraded, or OracleHealth::Broken
4. Quota Window Auto-Decision
When quota is exhausted with zero failures: breaker auto-closes (service healthy)
When quota is exhausted with any failures: breaker re-opens (service still degraded)
Window resets automatically after evaluation_window_s passes
Files Changed
File Changes
src/circuit_breaker.rs +4 public functions: probe_request(), half_open_probe_success(), half_open_probe_failure(), reset_half_open_window()
src/graceful_degradation.rs +1 public function: probe_oracle_with_circuit_breaker()
src/circuit_breaker_tests.rs +9 new tests covering all probe scenarios
Tests Added (9 new)
test_probe_request_admitted_within_quota — Quota-based admission works
test_probe_request_rejected_when_quota_exhausted — Rejection + auto-close (0 failures)
test_probe_request_cooldown_enforcement — Cooldown blocks probes
test_probe_request_not_half_open — Returns false in Closed/Open states
test_half_open_probe_success_tracks_completion — Auto-closes after threshold
test_half_open_probe_failure_reopens_circuit — Re-opens on single failure
test_probe_request_quota_window_resets — Window expiry + auto-close
test_quota_exhausted_with_failures_reopens — Quota full + failures → re-open
test_probe_oracle_with_circuit_breaker_integration — Full graceful degradation integration
Security & Design
Overflow-safe: All arithmetic uses saturating_add / saturating_sub
No unwrap(): All error paths use ? or match
Admin-only: State transitions (pause, resume) require AdminAccessControl validation
NatSpec docs: All new functions have /// rustdoc with parameters, returns, and behavior
Pre-existing Issue
Full cargo test is blocked by a pre-existing compilation error in market_id_generator.rs:773 (unclosed delimiter) that is unrelated to this PR's changes.
Closes #707