fix(orchestration): redrive lambda misses messages

[mrgrain]

The RedriveStateMachine lambda drains the backend DLQ by looping on ReceiveMessageCommand until it gets an empty response. It previously called ReceiveMessageCommand with no WaitTimeSeconds, which defaults to 0. SQS calls this short polling.

During failure recovery, I have observed that the Redrive-Lambda sometimes doesn't drain all messages. This fits with SQS's description of short polling, although I am not entirely sure how exactly. But because the loop exits as soon as result.Messages is empty, a single short-polled empty response ends the redrive early, leaving messages in the DLQ unprocessed.

Adding WaitTimeSeconds: 3 switches to long-polling: SQS waits up to 3s and samples more of the backend before responding, making an early empty result far less likely. Again, I have observed that this resolves the issue and all messages are now drained from the queue.

The PR also lowers VisibilityTimeout from 900s to 60s. The message is only deleted after a successful StartExecutionCommand, so the long timeout has no benefit on success. But it was super annoying on failure because every message was blocked for 15 minutes. Given that the Redrive-Lambda is manually executed, there's not much point in this.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[rix0rrr]

Conditionally approved.

I think WaitTimeSeconds should probably be the maximum to minimize the chances of race conditions. It's prooooooooobably fine with 3, but we also don't really have latency requirements so why not minimize them.

[mrgrain]

As discussed, there's no maximum wait time. It just needs to be lower than http configuration (which can be adjusted as well if needed).