HDDS-15824. Wait for pipeline before writing keys in debug replicas test#10722
Draft
smengcl wants to merge 1 commit into
Draft
HDDS-15824. Wait for pipeline before writing keys in debug replicas test#10722smengcl wants to merge 1 commit into
smengcl wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to reduce flakiness in the ozonesecure-ha debug replicas acceptance suite by ensuring write pipelines are re-established after a datanode restart before the Robot “Write keys” suite setup runs.
Changes:
- Add a bounded wait (180s) for at least one OPEN RATIS THREE pipeline before running
debug/ozone-debug-tests.robot. - Remove
JSONProviderfrom the httpfsgateway module (currently not described in the PR metadata and appears unrelated to HDDS-15824).
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| hadoop-ozone/dist/src/main/compose/common/replicas-test.sh | Adds an explicit wait for an OPEN RATIS THREE pipeline after datanode restart to avoid early key writes failing. |
| hadoop-ozone/httpfsgateway/src/main/java/org/apache/ozone/lib/wsrs/JSONProvider.java | Deletes the JSON MessageBodyWriter for JSONStreamAware (unexplained in PR description; likely unrelated to the acceptance-test flake). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
… debug-tools acceptance test The container.db backup flow in replicas-test.sh (added by HDDS-14225) stops all datanodes for a consistent backup, restarts them, and waits only for wait_for_datanode HEALTHY before running debug/ozone-debug-tests.robot. That wait confirms the SCM heartbeat state but not that write pipelines have been re-established, so the "Write keys" suite setup can fail with a non-zero exit (255) when it writes RATIS THREE and EC keys before a pipeline is available, causing the whole Ozone-Debug-Tests suite to fail as flaky. Wait for at least one OPEN RATIS THREE pipeline (bounded at 180s) after the datanode restart, before writing keys. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
smengcl
force-pushed
the
HDDS-15824-debug-tools-pipeline-wait
branch
from
70d924f to
8f8cc09
Compare
adoroszlai
reviewed
Jul 11, 2026
adoroszlai
left a comment
adoroszlai
left a comment
Contributor
There was a problem hiding this comment.
Thanks @smengcl for the patch.
Comment on lines
+74
to
+84
| echo "Waiting for an OPEN RATIS THREE pipeline before writing keys" | ||
| SECONDS=0 | ||
| until [[ "$(docker-compose exec -T ${SCM} bash -c \ | ||
| "ozone admin pipeline list --state OPEN --type RATIS --replication THREE --json" 2>/dev/null \ | ||
| | jq -r 'length')" -ge 1 ]]; do | ||
| if [[ $SECONDS -ge 180 ]]; then | ||
| echo "Timed out waiting for an OPEN RATIS THREE pipeline" >&2 | ||
| exit 1 | ||
| fi | ||
| sleep 2 | ||
| done |
Contributor
There was a problem hiding this comment.
Please add a function in testlib.sh for single check for pipeline (e.g. pipeline_exists), and use retry to repeat it.
RETRY_ATTEMPTS=60 retry pipeline_exists RATIS THREE
adoroszlai
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated-by: Claude Code (Opus 4.8, 1M context)
What changes were proposed in this pull request?
The
ozonesecure-hadebug replicas acceptance suite (Ozone-Debug-Tests :: Test ozone debug CLI) is flaky. ItsSuite Setup(Write keys) intermittently fails with255 != 0, which marks every test in the suite as failed even though no test logic ran.Root cause is in
hadoop-ozone/dist/src/main/compose/common/replicas-test.sh. The container.db backup flow introduced by HDDS-14225 (the RocksDB 7.7.3 to 10.10.1 upgrade, #9813) now stops all datanodes for a consistent backup, restarts them, and waits only forwait_for_datanode HEALTHYbefore runningdebug/ozone-debug-tests.robot.wait_for_datanodeconfirms the SCM heartbeat state (HEALTHY) but not that write pipelines have been re-established.Write keysthen writes a default RATIS THREE key, a RATIS ONE key, a RATIS THREE key, and an EC rs-3-2 key; if it runs before an OPEN pipeline is available after the mass restart, theozone sh key putexits non-zero (255) and the suite setup fails.Before HDDS-14225 the test backed up only
datanode1viadocker cpwithout stopping any datanode, and did not restart datanodes beforeWrite keys, so this race did not exist.Change
After the datanode restart and before running the debug test, wait (bounded at 180s) for at least one OPEN RATIS THREE pipeline via
ozone admin pipeline list --state OPEN --type RATIS --replication THREE --json. RATIS ONE and EC writes are covered by all five datanodes already beingHEALTHY; the RATIS THREE pipeline being OPEN is the gap this closes.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15824
How was this patch tested?
bash -npasses on the modified script. The change is exercised by the existingacceptance (tools)job, which runsozonesecure-ha/test-debug-tools.sh.Not reproduced locally (requires a full
-Pdistbuild plus a docker-compose acceptance run). Frequency was assessed from master history:test-debug-tools.shpassed in all 14 post-upgradeacceptance (tools)master runs (2026-07-08 to 07-10); the only archivedacceptance-toolsmaster failures in that window weretest-repair-tools.sh, a separate flake. The failure was observed on an unrelated PR run (2026-07-10) that does not touch this write path.