Skip to content

fix: retry message-less 400 on UploadPart - #147

Merged
ServerSideHannes merged 2 commits into
mainfrom
fix/retry-messageless-400-uploadpart
Jul 30, 2026
Merged

fix: retry message-less 400 on UploadPart#147
ServerSideHannes merged 2 commits into
mainfrom
fix/retry-messageless-400-uploadpart

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

Problem

Scylla backup runs keep failing on main.companies with:

multipart upload failed to upload part: InvalidArgument: InvalidArgument
	status code: 400

#143 added UploadPart retry and #144 made GatewayTimeout/504 retryable, but this variant still failed on first sight — one bad part out of thousands aborts the file, and through rclone's job error, the entire run. Observed 4x during a single run, on two of 13 racks.

Root cause

Hetzner sheds load under concurrency by returning a bare 400 with an empty body, which botocore surfaces as InvalidArgument with Message=None.

All 4 captured occurrences had Message=None, on legal 50 MiB non-final parts (parts 2–16, never part 1):

{"event": "UPLOAD_PART_CLIENT_ERROR", "part_num": 16,
 "content_length": 52428800,
 "aws_error_code": "InvalidArgument", "aws_error_message": null,
 "http_status": 400}

A real InvalidArgument always names the offending argument (ArgumentName/ArgumentValue). The message-less form is congestion, not a malformed request.

Reproduced: 24 × 50 MiB parts at 16-way concurrency against the production backend yielded 1 failure — surfaced as GatewayTimeout: "The server did not respond in time." Same congestion event, except Hetzner managed to write a body that time. Latency confirms the mechanism:

p50 max
unloaded (sequential) 1.79s 55.41s
16-way concurrent 13.09s 52.32s

A 7× latency degradation under load, with a heavy tail — so the backend starts dropping requests at the tail.

Fix

Treat a 400 with no message as retryable, alongside the existing transient set:

if code in ("InvalidArgument", "BadRequest", "400") and not err.get("Message"):
    return True

Deliberately narrow: a genuine InvalidArgument with a reason still fails immediately, so a truly malformed request can't retry forever.

Observability

The retry loop logged retries but nothing on the terminal outcome, so UPLOAD_PART_CLIENT_ERROR looked identical whether the part was rejected outright or gave up after 4 transient failures. That ambiguity has misdirected every previous investigation into this class of failure.

  • UPLOAD_PART_RECOVERED — part landed after N attempts (previously a silent success that hid how close a run ran to the edge)
  • UPLOAD_PART_GAVE_UPattempts, max_attempts, classified_retryable, exhausted, part_size, elapsed_sec, AWS code/message

UPLOAD_PART_RETRY also gains part_size, aws_error_code, max_attempts.

Testing

638 passed — 12 new tests covering the message-less 400 (retried), a real InvalidArgument with a reason (not retried), permanent errors, and both new log events.

Note: the log assertions target the logger directly rather than captured stdout — asserting on capsys passed in isolation but failed in the full suite, since another test reconfigures structlog.

Scope

This fixes one confirmed failure mode. Separately still open:

  • ContentLengthError truncated reads continue at ~5–30/min (handled by read_source_bytes resume from fix: retry 504 GatewayTimeout and resume truncated source reads #144, but the underlying congestion remains)
  • Scylla Manager has no retry above the job level (worker_upload.go:213 retries only errJobNotFound), so any error escaping the proxy still costs a whole host's upload
  • aws_error_message: null on a legal 50 MiB part is arguably a Hetzner bug worth reporting upstream

Hetzner sheds load under concurrency by returning a bare 400 with an empty
body, which botocore surfaces as InvalidArgument with Message=None. #144 made
GatewayTimeout retryable but not this variant, so it hit `raise` on first
sight and failed the whole multi-GB part -- and through rclone's job error,
the whole Scylla backup run.

All 4 occurrences captured on main.companies had Message=None, on legal 50MiB
non-final parts (part 2-16, never part 1). A real InvalidArgument always names
the offending argument, so the message-less form is congestion, not a
malformed request.

Reproduced: 24 x 50MiB parts at 16-way concurrency yielded 1 failure, surfaced
as GatewayTimeout("The server did not respond in time") -- the same congestion
event, with a body Hetzner managed to write. Latency degrades from p50 1.79s
unloaded to p50 13.09s / max 52.32s under concurrency.

Also adds two events the retry loop was missing:

  UPLOAD_PART_RECOVERED  a part that only landed after N attempts, previously
                         a silent success that hid how close a run ran to the
                         edge
  UPLOAD_PART_GAVE_UP    carries attempts/max_attempts/classified_retryable/
                         exhausted, so "gave up after N transient failures" is
                         distinguishable from "rejected outright" -- they were
                         identical in UPLOAD_PART_CLIENT_ERROR, which made
                         every past investigation guess between the two
@ServerSideHannes
ServerSideHannes merged commit 25732b5 into main Jul 30, 2026
11 checks passed
@ServerSideHannes
ServerSideHannes deleted the fix/retry-messageless-400-uploadpart branch July 30, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant