in_s3: stop retrying SQS messages that can never succeed - #484
Merged
Watson1978 merged 1 commit intoAug 19, 2026
Merged
Conversation
Watson1978
marked this pull request as draft
August 17, 2026 06:03
Watson1978
force-pushed
the
in_s3-validate-sqs-message-structure
branch
3 times, most recently
from
August 17, 2026 08:26
fdc9195 to
22c6239
Compare
Watson1978
force-pushed
the
in_s3-validate-sqs-message-structure
branch
2 times, most recently
from
August 18, 2026 01:15
8ea2375 to
aa20d44
Compare
Watson1978
force-pushed
the
in_s3-validate-sqs-message-structure
branch
3 times, most recently
from
August 18, 2026 02:12
68d31be to
41f6477
Compare
Watson1978
marked this pull request as ready for review
August 18, 2026 02:27
kenhys
reviewed
Aug 18, 2026
The rescue around each message turned every failure into :skip_delete, so an error that recurs on redelivery kept the message in the queue until the retention period expired. Posting "hello" to the queue was enough to stall a source for up to 14 days, and a corrupt or unparsable object did the same once fetched: nothing is ever ingested and a backtrace is logged on every pass. skip_delete is false by default, so anyone who can put a message on the queue reaches it. - Name the errors no redelivery can fix, a body that is not JSON, an archive zlib rejects, bytes the parser cannot read, and delete those messages rather than keep them. The list is our own: it overlaps with the one Fluent::Plugin::Output keeps for its retries, but an input plugin should not depend on where an output plugin draws that line - Leave NoMethodError out of it, since it is more likely a defect of ours than a property of the message, and reach the object key with dig so that a malformed notification yields nil rather than raising it - Keep every other failure on :skip_delete, so a networking error, a throttle or a missing object is still retried - Name the object key in the warn that deletes, truncated and scrubbed, because that line is the only record the object leaves behind, and log the backtrace there too, since our own bugs reach the same path - Raise an unrecoverable CommandError when an extractor command exits non-zero, but a plain error when it was signalled, and let it out of the lzo, lzma2 and gzip wrappers, whose rescue named an out-of-scope path and raised NameError over the real cause - Report a message that is not an S3 notification instead of dropping it in silence, and document in docs/input.md which failures delete Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
Watson1978
force-pushed
the
in_s3-validate-sqs-message-structure
branch
from
August 18, 2026 05:56
41f6477 to
1f7d9a2
Compare
kenhys
approved these changes
Aug 18, 2026
Contributor
Author
|
Thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Posting
helloto the queue is enough to stall anin_s3source for up to 14 days. Every per-message failure becomes:skip_delete, so SQS redelivers the message after the visibility timeout, it fails identically, and this repeats until the retention period expires — burning CPU, log volume and SQS requests with nothing ever ingested.skip_deleteisfalseby default, so anything that can put a message on the queue can trigger it, and a corrupt or unparsable object does the same once fetched.zlibrejects, bytes the parser cannot read — and delete such a message instead of retrying it forever. The list is the plugin's own: it overlaps with the oneFluent::Plugin::Outputkeeps for its retries, but an input plugin should not depend on where an output plugin draws that lineNoMethodErrorout of that list: it is more likely a defect of ours than a property of the message, and stalling the queue beats turning our own bug into lost notifications.get_raw_keynow usesdigso a malformed body yieldsnilrather than raising it:skip_delete, so a networking error, a throttle or a missing object is still retried, and a redrive policy with a dead letter queue stays the tool for capping thatCommandErrorwhen an extractor command exits non-zero, but a plain error when the process was signalled, and let it out of the lzo, lzma2 and gzip wrappers, whose"Failed to extract #{path}"named a local ofextract_with_commandand raisedNameErrorover the real causedocs/input.mdwhich failures delete the message, so that a conservativedecompression_size_limitis not chosen without knowing it discards notificationsin_s3tests go from 46 to 81, covering both sides of the boundary and locked with a mutation pass. Parser exceptions outsideMessagePack::UnpackError, such asCSV::MalformedCSVError, are left for a separate change. Two pre-existing limitations are unchanged: onlyRecords[0]is examined, andadd_object_metadatareports the URL-escaped key.