HDDS-15802. Add integration test for per-volume push replication thread pools on DataNode by jojochuang · Pull Request #10703 · apache/ozone

jojochuang · 2026-07-09T23:01:23Z

What changes were proposed in this pull request?

HDDS-15802. Add integration/system tests to certify per-volume push replication thread pools on DataNode (HDDS-15412).

Please describe your PR in detail:

P0 tests:

Priority	Test	Description
P0	Push/pull regression with per-volume enabled	Multi-volume DN (setNumDataVolumes(2)), feature ON. Verify existing push/pull replication from TestContainerReplication still succeeds. Assert volume pools are created (count matches DN volume count).
P0	Volume failure: healthy-volume replication continues	Fail one volume on a multi-volume DN while per-volume pools are enabled. Assert failed-volume pool is removed; push replication for containers on healthy volumes still succeeds; DN remains within toleration limits.
P0	Decommission E2E with per-volume pools	Multi-volume DN with Ratis and/or EC data; feature ON with streams.limit=1. Decommission the DN. Assert node reaches DECOMMISSIONED, under-replicated containers recover, and cluster remains healthy.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15802

How was this patch tested?

Test only

Introduce optional per-disk replication handler thread pools so slow push replication on one volume does not block replication on other disks. Pull replication was removed upstream; EC/reconcile tasks continue using the global pool. Includes runtime resize support and volume-failure cleanup. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I7cacd6fbeb435f5172fc67141a869cd4d1073811

Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I87c4c544a24f99675ef68940b9412367d60c9b8e

Certify push/SCM replication, volume failure continuity, and decommission E2E behavior with per-volume replication thread pools enabled on DataNode. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I6060b2d3bf40a710b13db882fcd3ca999756d63f

adoroszlai · 2026-07-10T11:22:25Z

+  @Test
+  void testPushAndScmReplicationWithPerVolumeEnabled() throws Exception {
+    OzoneConfiguration conf = createPerVolumeConfig(true, 1);
+    try (MiniOzoneCluster cluster = newCluster(conf, 3);


Please don't start new cluster for each test case.

[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.7 s -- in org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication

The HDDS-15082 feature exposes hdds.datanode.replication.per.volume.streams.limit as reconfigurable; update the expected property set so integration CI passes. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Ib775c19283e6b7e0a4acf861dfcbcad2af4dd37d

Copilot

Pull request overview

Adds per-volume push replication handler thread pools on the DataNode, wires them into the replication supervisor lifecycle (including volume-failure handling and runtime resizing), and introduces new unit/integration coverage plus reconfigurability documentation for the new configuration knobs.

Changes:

Introduce per-volume push-replication executor pools and route push replication tasks to a volume-specific pool when enabled.
Add runtime reconfiguration support for the per-volume streams limit, plus documentation and reconfig property listing updates.
Add unit tests around supervisor behavior and new integration tests validating per-volume pool behavior under normal replication, disk failure, and decommission scenarios.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/reconfig/TestDatanodeReconfiguration.java	Adds per-volume streams limit key to the list of expected DN reconfigurable properties.
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/replication/TestPerVolumePushReplication.java	New integration tests for per-volume push replication pools, volume failure behavior, and decommission flows.
hadoop-hdds/docs/content/feature/Reconfigurability.md	Documents the new reconfigurable per-volume streams limit property.
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationSupervisor.java	Adds unit coverage for per-volume pool initialization, sizing/resizing, isolation, and failure handling.
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestReplicationConfig.java	Adds tests for new per-volume replication config defaults/validation and parsing.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java	Registers and implements reconfig handler to resize per-volume pools at runtime.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/VolumeReplicationThreadPools.java	New implementation managing per-volume ThreadPoolExecutors and resizing/shutdown behavior.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationTask.java	Adds helper to classify push replication tasks for executor selection logic.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisor.java	Adds per-volume executor selection, pool lifecycle management, queue sizing changes, and new supervisor APIs.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/ReplicationServer.java	Introduces new config keys for per-volume enablement and per-volume streams limit (with validation).
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java	Wires containerSet/volumeSet into supervisor and hooks volume-failure listener to shut down failed volume pools.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+      decrementTaskCounter(task);
+      return;
+    }
+    taskExecutor.execute(new TaskRunner(task));


+      VolumeReplicationThreadPools pools =
+          sourceDn.getDatanodeStateMachine().getSupervisor().getVolumeReplicationThreadPools();
+      assertNotNull(pools);
+      assertFalse(pools.hasPool(vol0.getStorageDir().getPath()));
+      assertTrue(pools.hasPool(vol1.getStorageDir().getPath()));


Use one cluster for all three integration tests per review feedback. Wait for per-volume pool removal after volume failure before asserting, and resolve containers by key for the decommission scenario on a shared cluster. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I4059d6bf45ec8a21271dea93ceb888ab80177194

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

+    ExecutorService taskExecutor = selectExecutor(task);
+    if (taskExecutor == null) {
+      LOG.warn("Rejected {} in ReplicationSupervisor: no replication handler "
+          + "thread pool available", task);
+      queuedCounter.get(task.getMetricName()).decrementAndGet();
+      inFlight.remove(task);
+      decrementTaskCounter(task);
+      return;
+    }
+    taskExecutor.execute(new TaskRunner(task));
+  }


+  /**
+   * @return true if this is a push replication task (source datanode sends
+   *     container to target).
+   */
+  public boolean isPushReplication() {
+    return cmd.getTargetDatanode() != null;
+  }


…082-p0 Merge apache/master to pick up latest fixes. Resolve TestReplicationSupervisor import conflict (TestClock -> MockClock). Wait for SCM pipelines in TestDnRatisLogParser before accessing the first pipeline. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I9e02d71045496c8f1a910e0b5cee74a487a2dc79

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

+    String volumeLabel = volumeRoot.substring(
+        Math.max(0, volumeRoot.lastIndexOf('/') + 1));
+    ThreadFactory threadFactory = new ThreadFactoryBuilder()
+        .setDaemon(true)
+        .setNameFormat(threadNamePrefix + "ContainerReplicationThread-"
+            + volumeLabel + "-%d")
+        .build();


+    StateContext context = stateMachine.getContext();
+    context.getTermOfLeaderSCM().ifPresent(cmd::setTerm);
+    context.addCommand(cmd);
+    waitFor(() -> counter.applyAsLong(supervisor) == previousCount + 1, 100, 30000);


+    assertEquals(previousFailures + 1,
+        sourceDn.getDatanodeStateMachine().getSupervisor().getReplicationFailureCount());


The pipeline wait belongs in a separate Jira/PR off master. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I302e7812d6db6862c690ccfc34980353a157ee64

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

+    if (taskExecutor == null) {
+      LOG.warn("Rejected {} in ReplicationSupervisor: no replication handler "
+          + "thread pool available", task);
+      queuedCounter.get(task.getMetricName()).decrementAndGet();
+      inFlight.remove(task);
+      decrementTaskCounter(task);
+      return;
+    }


+    container.getVolumeSet().setFailedVolumeListener(() -> {
+      container.handleVolumeFailures();
+      supervisor.shutdownFailedVolumePools(container.getVolumeSet());
+    });


+    } finally {
+      supervisor.stop();
+    }


@@ -224,6 +230,29 @@ public static final class ReplicationConfig {
    )
    private double outOfServiceFactor = OUTOFSERVICE_FACTOR_DEFAULT;

+    @Config(key = PER_VOLUME_ENABLED_KEY,
+        type = ConfigType.BOOLEAN,
+        defaultValue = "false",
+        tags = {DATANODE},
+        description = "When true, push-based container replication uses a " +
+            "separate replication handler thread pool per data volume so " +
+            "that slow replication on one disk does not block replication " +
+            "on other disks. Pull replication and other replication tasks " +
+            "continue to use the global replication handler thread pool."
+    )
+    private boolean perVolumeEnabled = false;
+
+    @Config(key = PER_VOLUME_STREAMS_LIMIT_KEY,
+        type = ConfigType.INT,
+        defaultValue = "1",
+        reconfigurable = true,
+        tags = {DATANODE},
+        description = "The maximum number of concurrent push replication " +
+            "commands per data volume when per-volume replication thread " +
+            "pools are enabled."
+    )
+    private int perVolumeStreamsLimit = PER_VOLUME_STREAMS_LIMIT_DEFAULT;


jojochuang and others added 3 commits July 8, 2026 15:57

HDDS-15412. Fix checkstyle violations.

320eacd

Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I87c4c544a24f99675ef68940b9412367d60c9b8e

adoroszlai reviewed Jul 10, 2026

View reviewed changes

adoroszlai changed the title ~~HDDS-15802. Add integration/system tests to certify per-volume push replication thread pools on DataNode (HDDS-15412).~~ HDDS-15802. Add integration test for per-volume push replication thread pools on DataNode Jul 10, 2026

Copilot AI review requested due to automatic review settings July 10, 2026 17:31

Copilot started reviewing on behalf of jojochuang July 10, 2026 17:35 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

adoroszlai added the test label Jul 10, 2026

Copilot AI review requested due to automatic review settings July 10, 2026 20:36

Copilot started reviewing on behalf of jojochuang July 10, 2026 20:36 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 22:10

Copilot started reviewing on behalf of jojochuang July 10, 2026 22:10 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

HDDS-15802. Revert unrelated TestDnRatisLogParser fix from p0 branch.

e4a3ae7

The pipeline wait belongs in a separate Jira/PR off master. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I302e7812d6db6862c690ccfc34980353a157ee64

Copilot AI review requested due to automatic review settings July 10, 2026 22:15

Copilot started reviewing on behalf of jojochuang July 10, 2026 22:15 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

jojochuang added the AI-gen label Jul 11, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-15802. Add integration test for per-volume push replication thread pools on DataNode#10703

HDDS-15802. Add integration test for per-volume push replication thread pools on DataNode#10703
jojochuang wants to merge 7 commits into
apache:masterfrom
jojochuang:HDDS-15082-p0

jojochuang commented Jul 9, 2026

Uh oh!

adoroszlai Jul 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		assertEquals(previousFailures + 1,
		sourceDn.getDatanodeStateMachine().getSupervisor().getReplicationFailureCount());

Uh oh!

Conversation

jojochuang commented Jul 9, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

adoroszlai Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants