Skip to content

in_s3: stop retrying SQS messages that can never succeed - #484

Merged
Watson1978 merged 1 commit into
fluent:masterfrom
Watson1978:in_s3-validate-sqs-message-structure
Aug 19, 2026
Merged

in_s3: stop retrying SQS messages that can never succeed#484
Watson1978 merged 1 commit into
fluent:masterfrom
Watson1978:in_s3-validate-sqs-message-structure

Conversation

@Watson1978

@Watson1978 Watson1978 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Posting hello to the queue is enough to stall an in_s3 source for up to 14 days. Every per-message failure becomes :skip_delete, so SQS redelivers the message after the visibility timeout, it fails identically, and this repeats until the retention period expires — burning CPU, log volume and SQS requests with nothing ever ingested. skip_delete is false by default, so anything that can put a message on the queue can trigger it, and a corrupt or unparsable object does the same once fetched.

  • Name the errors that no redelivery can fix — a body that is not JSON, an archive zlib rejects, bytes the parser cannot read — and delete such a message instead of retrying it forever. The list is the plugin's own: it overlaps with the one Fluent::Plugin::Output keeps for its retries, but an input plugin should not depend on where an output plugin draws that line
  • Leave NoMethodError out of that list: it is more likely a defect of ours than a property of the message, and stalling the queue beats turning our own bug into lost notifications. get_raw_key now uses dig so a malformed body yields nil rather than raising it
  • Keep every other failure on :skip_delete, so a networking error, a throttle or a missing object is still retried, and a redrive policy with a dead letter queue stays the tool for capping that
  • Name the object key in the warn that deletes, truncated to 256 bytes and scrubbed, because that line is the only record the object leaves behind, and log the backtrace there too since our own bugs reach the same path
  • Raise an unrecoverable CommandError when an extractor command exits non-zero, but a plain error when the process was signalled, and let it out of the lzo, lzma2 and gzip wrappers, whose "Failed to extract #{path}" named a local of extract_with_command and raised NameError over the real cause
  • Document in docs/input.md which failures delete the message, so that a conservative decompression_size_limit is not chosen without knowing it discards notifications

in_s3 tests go from 46 to 81, covering both sides of the boundary and locked with a mutation pass. Parser exceptions outside MessagePack::UnpackError, such as CSV::MalformedCSVError, are left for a separate change. Two pre-existing limitations are unchanged: only Records[0] is examined, and add_object_metadata reports the URL-escaped key.

@Watson1978
Watson1978 marked this pull request as draft August 17, 2026 06:03
@Watson1978 Watson1978 changed the title in_s3: validate SQS message structure before processing in_s3: stop retrying SQS messages that can never succeed Aug 17, 2026
@Watson1978
Watson1978 force-pushed the in_s3-validate-sqs-message-structure branch 3 times, most recently from fdc9195 to 22c6239 Compare August 17, 2026 08:26
@Watson1978 Watson1978 changed the title in_s3: stop retrying SQS messages that can never succeed in_s3: stop retrying SQS messages that cannot be decoded Aug 17, 2026
@Watson1978
Watson1978 force-pushed the in_s3-validate-sqs-message-structure branch 2 times, most recently from 8ea2375 to aa20d44 Compare August 18, 2026 01:15
@Watson1978 Watson1978 changed the title in_s3: stop retrying SQS messages that cannot be decoded in_s3: stop retrying SQS messages that can never succeed Aug 18, 2026
@Watson1978
Watson1978 force-pushed the in_s3-validate-sqs-message-structure branch 3 times, most recently from 68d31be to 41f6477 Compare August 18, 2026 02:12
@Watson1978
Watson1978 marked this pull request as ready for review August 18, 2026 02:27
@Watson1978
Watson1978 requested a review from kenhys August 18, 2026 02:27
Comment thread lib/fluent/plugin/in_s3.rb
The rescue around each message turned every failure into :skip_delete, so
an error that recurs on redelivery kept the message in the queue until the
retention period expired. Posting "hello" to the queue was enough to stall
a source for up to 14 days, and a corrupt or unparsable object did the same
once fetched: nothing is ever ingested and a backtrace is logged on every
pass. skip_delete is false by default, so anyone who can put a message on
the queue reaches it.

- Name the errors no redelivery can fix, a body that is not JSON, an archive
  zlib rejects, bytes the parser cannot read, and delete those messages
  rather than keep them. The list is our own: it overlaps with the one
  Fluent::Plugin::Output keeps for its retries, but an input plugin should
  not depend on where an output plugin draws that line
- Leave NoMethodError out of it, since it is more likely a defect of ours
  than a property of the message, and reach the object key with dig so that
  a malformed notification yields nil rather than raising it
- Keep every other failure on :skip_delete, so a networking error, a
  throttle or a missing object is still retried
- Name the object key in the warn that deletes, truncated and scrubbed,
  because that line is the only record the object leaves behind, and log
  the backtrace there too, since our own bugs reach the same path
- Raise an unrecoverable CommandError when an extractor command exits
  non-zero, but a plain error when it was signalled, and let it out of the
  lzo, lzma2 and gzip wrappers, whose rescue named an out-of-scope path
  and raised NameError over the real cause
- Report a message that is not an S3 notification instead of dropping it
  in silence, and document in docs/input.md which failures delete

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978
Watson1978 force-pushed the in_s3-validate-sqs-message-structure branch from 41f6477 to 1f7d9a2 Compare August 18, 2026 05:56
@Watson1978

Copy link
Copy Markdown
Contributor Author

Thanks

@Watson1978
Watson1978 merged commit bf42439 into fluent:master Aug 19, 2026
9 checks passed
@Watson1978
Watson1978 deleted the in_s3-validate-sqs-message-structure branch August 19, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants