Back off between failed connection attempts in the HTTP outlet - #50
Merged
Merged
Conversation
The HTTP device's outlet thread reconnects immediately after a refused or dropped connection, which spins a CPU core at thousands of connection attempts per second while the ingesting host is unreachable. These tests pin the intended behaviour: - after each failed connection the outlet waits 1, 2, 4, 8, 16, then 30 seconds before reconnecting; - a request failing on an open connection waits the same way and is still dropped after 3 attempts; - a delivered request starts the wait over at 1 second; - close stops the outlet at once while it waits. A config spec pins that reading the unset debug logger does not warn under ruby -w (Ruby before 3.0), which the outlet does on every attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the ingesting host refused or dropped the connection, the outlet thread reconnected at once, in a busy loop: thousands of connection attempts per second on one CPU core for as long as the host stayed unreachable. The outlet now waits before reconnecting after a failed connection, be it an exception from Net::HTTP#start or a request that failed on an open connection (deliver_requests returning false): 1 second, doubling on every consecutive failure up to 30 seconds. A delivered request starts the wait over at 1 second. Requests are still re-queued and dropped after 3 attempts. close kills the outlet thread, which interrupts the sleep, so it does not wait for the backoff. Logtail::Config initialises @debug_logger to nil, so reading the unset debug logger, which the outlet does on every attempt, no longer warns under ruby -w on Ruby before 3.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cold TruffleRuby can take a while to run the new thread's first connection attempt; the 1 s poll left little headroom on CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the ingesting host refuses or drops the connection, the HTTP device's outlet thread reconnects immediately, in a tight loop. Against a refused port that is thousands of connection attempts per second on one CPU core for as long as the host stays unreachable, and on Ruby before 3.0 under
ruby -wevery attempt also printsinstance variable @debug_logger not initializedto stderr.This PR makes the outlet wait before reconnecting after a failed connection:
Logtail::Configalso initialises@debug_loggertonil, so reading the unset debug logger no longer warns underruby -w.The first commit contains only the new tests and is expected to fail on CI; the second commit makes them pass, and the third gives the thread-based spec more time for its first attempt on a cold TruffleRuby.
End to end in
ruby:4.0-slim, with the gem from this branch (Bundler git source at 2bd7a73) against released 0.1.19:127.0.0.1:9), one log line, 20 s: 5 connection attempts at 0, 1, 3, 7 and 15 s and 0.01 s of CPU, against 110,809 attempts and 7.3 s of CPU with 0.1.19;🤖 Generated with Claude Code