[CELEBORN-2415] Add Celeborn Spark UI extension by yew1eb · Pull Request #3793 · apache/celeborn

yew1eb · 2026-08-10T10:17:44Z

What changes were proposed in this pull request?

Add a new module celeborn-client-spark-3-ui — a Spark UI extension that renders a "Celeborn" tab in both the live Spark UI and the History Server (enabled via spark.plugins=org.apache.spark.shuffle.celeborn.CelebornPlugin, supports Spark 3.x and 4.x).

This is an initial version that lays the foundation; it intentionally starts minimal. The page shows:

Summary — shuffle write/read bytes, time, speed, and shuffle-vs-task duration ratio, aggregated from Spark's built-in TaskMetrics
Celeborn Properties — a collapsible table of spark.celeborn.* configs

The skeleton is: CelebornPlugin (live UI) / CelebornHistoryServerPlugin (SHS, via AppHistoryServerPlugin SPI) → listener aggregates into Spark KVStore → CelebornStatusStore → CelebornUITab renders; a javax/jakarta servlet bridge covers Spark 3.x and 4.x. This plugin/store/render pipeline is designed so that fine-grained Celeborn client metrics can be added as collapsible sections in follow-up PRs:

Shuffle Write Times — serialize / copy / queue wait / compress / queue stall / inflight wait / drain wait / max push RTT / slow push
Shuffle Read Times — chunk wait / decompress / retry wait / max chunk RTT / slow chunk
Shuffle Write Servers — per-worker push bytes, push count, RTT, soft/hard splits, congested counts, last failure reason
Shuffle Read Servers — per-worker chunk count, read bytes, total/max RTT
Shuffle Assignments — partition-to-worker slot assignments

Why are the changes needed?

Spark UI already shows stage/task-level shuffle metrics, but nothing Celeborn-specific: which workers served the data, push/chunk latencies, congestion and split events are invisible there, while Celeborn's own metrics sit in cluster-level Grafana dashboards that cannot be attributed back to a specific application. This tab closes that gap by putting per-application Celeborn shuffle information inside the Spark UI and History Server, alongside the job's stages and tasks.

Does this PR introduce any user-facing change?

Yes.
Setting spark.plugins=org.apache.spark.shuffle.celeborn.CelebornPlugin (with the extension jar on the classpath) adds a "Celeborn" tab to the Spark UI and History Server. Disabled by default; no effect otherwise.

Does this PR resolve a correctness bug?

No.

How was this patch tested?

Manual end-to-end verification:

yew1eb · 2026-08-11T03:56:10Z

Ping @pan3793 @cxzl25 @RexXiong @SteNicholas @zaynt4606

Copilot

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds a new Spark UI / History Server extension module that surfaces Celeborn-related shuffle summary metrics and spark.celeborn.* configuration in a dedicated “Celeborn” tab.

Changes:

Introduces new module client-spark/spark-3-ui with Spark plugin + History Server plugin, status-store access, and UI rendering.
Adds a Spark 3 vs Spark 4 servlet (javax vs jakarta) bridge via profile-selected source roots.
Wires the new UI module into root build and shaded Spark client artifacts.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pom.xml	Adds the new UI module to Spark build profiles and introduces `servlet.source.dir` to select Spark3/Spark4 servlet bridge sources.
client-spark/spark-3-ui/pom.xml	New module POM with Spark dependency + servlet APIs and build-helper source-root selection.
client-spark/spark-4-shaded/pom.xml	Includes the new UI artifact in the Spark 4 shaded assembly dependencies.
client-spark/spark-3-shaded/pom.xml	Includes the new UI artifact in the Spark 3 shaded assembly dependencies.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornPlugin.scala	Adds SparkPlugin entrypoint for live UI, registers listener and attaches tab.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornHistoryServerPlugin.scala	Adds SHS plugin SPI implementation to replay metrics and attach the tab.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornListener.scala	Listener aggregates task metrics and persists rollups + properties into KVStore.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornStatusStore.scala	Status-store accessors for aggregated metrics and celeborn properties.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/ui/CelebornUITab.scala	Defines the “Celeborn” tab and registers the page.
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/ui/CelebornShufflePage.scala	Renders summary metrics + collapsible properties table.
client-spark/spark-3-ui/src/main/scala-spark3/org/apache/spark/shuffle/celeborn/ui/SparkServletBridge.scala	Spark 3 servlet type alias (`javax.servlet`).
client-spark/spark-3-ui/src/main/scala-spark4/org/apache/spark/shuffle/celeborn/ui/SparkServletBridge.scala	Spark 4 servlet type alias (`jakarta.servlet`).
client-spark/spark-3-ui/src/main/resources/META-INF/services/org.apache.spark.status.AppHistoryServerPlugin	Registers the SHS plugin via SPI.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

yew1eb · 2026-08-18T11:52:52Z

+private[celeborn] class CelebornPropertiesUIData(
+    val info: Seq[(String, String)]) {
+
+  @JsonIgnore
+  @KVIndex
+  def id: String = classOf[CelebornPropertiesUIData].getName
+}


Not necessary. Verified end-to-end that the properties table renders correctly in both live UI and SHS replay. Spark's KVStore uses Jackson with the Scala

yew1eb · 2026-08-18T11:55:33Z

+    if (statusStore.hasData()) {
+      new CelebornUITab(statusStore, ui)
+    }


Fair point on the live/SHS inconsistency. Fixed by attaching the tab unconditionally in SHS (matching the live UI, which always attaches once spark.plugins=CelebornPlugin is set) and removing the now-unused hasData().

yew1eb · 2026-08-18T11:56:06Z

+  private val totalTaskDurationMs = new AtomicLong(0L)
+
+  private val lastUpdateTimestamp = new AtomicLong(-1L)
+  private val updateIntervalMillis = 5000L


Reasonable for later, but intentionally a fixed default for the MVP. We can expose spark.celeborn.ui.flushIntervalMs in a follow-up once we see real cadence
needs.

yew1eb · 2026-08-18T11:57:00Z

+    val last = lastUpdateTimestamp.get()
+    if (!force && (last != -1L && (now - last) < updateIntervalMillis)) {
+      return
+    }
+    if (lastUpdateTimestamp.compareAndSet(last, now) || force) {
+      flushAggregations()
+    }


Impact is negligible in practice: force=true is only reached from onApplicationEnd, which fires once at app end with no concurrent onTaskEnd. No extra flushes follow it. Leaving as-is.

yew1eb · 2026-08-18T11:57:45Z

+    <dependency>
+      <groupId>jakarta.servlet</groupId>
+      <artifactId>jakarta.servlet-api</artifactId>
+      <scope>provided</scope>
+    </dependency>


Intentional. A single module pom with ${servlet.source.dir} selecting the source root keeps the build simple; both servlet APIs are provided and only the
one matching the selected source root is ever referenced at compile time. Splitting into per-profile dependencies would duplicate the dependency block
across every Spark profile without real benefit.

yew1eb · 2026-08-18T11:58:25Z

        <module>client-spark/spark-3</module>
        <module>client-spark/spark-3-columnar-common</module>
        <module>client-spark/spark-3-columnar-shuffle</module>
+        <module>client-spark/spark-3-ui</module>


The name follows the existing celeborn-client-spark-3 convention in this repo, which also serves Spark 4. Renaming would diverge from that. Keeping spark-3-ui for consistency.

yew1eb · 2026-08-18T11:59:19Z

+        <span class="collapse-aggregated-celebornProperties collapse-table"
+            onClick="collapseTable('collapse-aggregated-celebornProperties',
+            'aggregated-celebornProperties')">
+          <h4>
+            <span class="collapse-table-arrow arrow-open"></span>
+            <a>Celeborn Properties</a>
+          </h4>
+        </span>
+        <div class="aggregated-celebornProperties collapsible-table">


Following Spark's own UI convention here — Spark uses camelCase collapse identifiers (e.g. collapse-aggregated-allActiveStages). The collapseTable() JS
accepts arbitrary ids as long as the collapse-* prefix and the target class match, which they do. No change.

setupUI() no longer gates on hasData(); the Celeborn tab is now always attached in the History Server, matching the live UI behavior where the tab is attached unconditionally once spark.plugins=CelebornPlugin is set. Removes the now-unused CelebornStatusStore.hasData().

zaynt4606 · 2026-08-19T10:09:48Z

ping @Kalvin2077

github-actions Bot added module:client module:spark kind:build labels Aug 10, 2026

yew1eb marked this pull request as draft August 10, 2026 11:30

yew1eb marked this pull request as ready for review August 10, 2026 13:38

yew1eb force-pushed the celeborn-ui-mvp branch 2 times, most recently from 4f8bad3 to 68fc2f2 Compare August 10, 2026 13:40

yew1eb marked this pull request as draft August 10, 2026 13:48

yew1eb force-pushed the celeborn-ui-mvp branch 6 times, most recently from 7a9e389 to 47f3676 Compare August 10, 2026 16:27

yew1eb closed this Aug 10, 2026

yew1eb reopened this Aug 10, 2026

[CELEBORN-2415][FOLLOWUP] Add Celeborn Spark UI extension

e053012

yew1eb force-pushed the celeborn-ui-mvp branch from 47f3676 to e053012 Compare August 10, 2026 16:57

yew1eb marked this pull request as ready for review August 10, 2026 17:30

cxzl25 requested a lite review from Copilot August 18, 2026 10:34

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Copilot started reviewing on behalf of cxzl25 August 18, 2026 11:03 View session

yew1eb force-pushed the celeborn-ui-mvp branch from c0cd150 to 18cf1d1 Compare August 18, 2026 12:08

yew1eb closed this Aug 18, 2026

yew1eb reopened this Aug 18, 2026

Navigation Menu

Uh oh!

[CELEBORN-2415] Add Celeborn Spark UI extension - #3793

[CELEBORN-2415] Add Celeborn Spark UI extension#3793
yew1eb wants to merge 2 commits into
apache:mainfrom
yew1eb:celeborn-ui-mvp

yew1eb commented Aug 10, 2026 •

edited

Loading

Uh oh!

yew1eb commented Aug 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

yew1eb Aug 18, 2026

Uh oh!

yew1eb Aug 18, 2026 •

edited

Loading

Uh oh!

yew1eb Aug 18, 2026

Uh oh!

yew1eb Aug 18, 2026

Uh oh!

yew1eb Aug 18, 2026

Uh oh!

yew1eb Aug 18, 2026

Uh oh!

yew1eb Aug 18, 2026

Uh oh!

zaynt4606 commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

yew1eb commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

Does this PR resolve a correctness bug?

How was this patch tested?

Uh oh!

yew1eb commented Aug 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

yew1eb Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

zaynt4606 commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yew1eb commented Aug 10, 2026 •

edited

Loading

yew1eb Aug 18, 2026 •

edited

Loading