[CELEBORN-2390] Add docker-compose for local cluster by yew1eb · Pull Request #3768 · apache/celeborn

yew1eb · 2026-07-25T07:21:25Z

What changes were proposed in this pull request?

Add a docker-compose.yaml under docker/ to run a minimal local Celeborn cluster (1 master + 3 workers) for development and debugging, with a build-image.sh wrapper and a conf/. The existing docker/Dockerfile is unchanged.

Why are the changes needed?

There is no lightweight local deployment path today — only the K8s Helm chart or running master/worker manually on the host.

Does this PR resolve a correctness bug?

Yes

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

Ran the cluster locally with docker compose up -d:

$ docker compose -f docker/docker-compose.yaml up -d
$ docker compose -f docker/docker-compose.yaml ps

yew1eb · 2026-07-31T09:37:32Z

@SteNicholas @RexXiong @zaynt4606 Could you help review this PR? Thanks!

yew1eb · 2026-08-04T03:06:51Z

Ping @pan3793, @cxzl25, @RexXiong, @SteNicholas

pan3793 · 2026-08-13T08:29:49Z

I feel this might be less useful without Compute Engine integration ... if you suppose run Compute Engine outside, e.g., run Spark on your host with local mode, then it requires a direct connection from Spark to Celeborn Worker, you should expose the Worker's port too

yew1eb · 2026-08-13T09:48:02Z

Thanks @pan3793. Added an example Spark cluster (1 master + 2 workers) to docker-compose so the stack can be exercised end-to-end.
Spark runs inside celeborn-net and contacts the master directly via spark.celeborn.master.endpoints=celeborn-master:9097, so no worker ports need to be exposed — worker push/fetch ports stay random, handed out by the master.

./docker/build-image.sh -Pspark-3.5      # celeborn:dev + celeborn-spark:dev
docker compose -f docker/docker-compose.yaml up -d
docker exec celeborn-spark-master /opt/spark/bin/spark-submit \
    --master spark://celeborn-spark-master:7077 \
    --class org.apache.spark.examples.GroupByTest \
    /opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar 10 100 1000 10

Reviewer noted the standalone Celeborn compose is less useful without a compute engine: it cannot be exercised end-to-end, and running Spark in local mode on the host would require exposing worker ports. Add an example Spark cluster (1 master + 2 workers) to docker-compose whose shuffle is served by Celeborn, addressing both concerns: - Spark runs inside the celeborn-net, so the client only contacts the master RPC (celeborn-master:9097); worker push/fetch ports stay random and unexposed. - A GroupByTest spark-submit example gives an end-to-end shuffle check. Changes: - docker/spark/Dockerfile: apache/spark:3.5.0 + celeborn client shaded jar - docker/spark/conf/spark-defaults.conf: CelebornShuffleManager/Kryo/... - docker-compose.yaml: spark-master/spark-worker services + shared volume - build-image.sh: optionally build celeborn-spark:dev when -Pspark-3.x produces dist/spark/*.jar; base tag via CELEBORN_SPARK_BASE_TAG (renamed to avoid collision with Spark's own SPARK_IMAGE_TAG env) - README.md: Spark build + GroupByTest run/verify instructions - .gitignore: exclude any jar copied into the docker/spark build context

pan3793

couples of minor issues, overall lgtm

pan3793 · 2026-08-20T14:59:53Z

 tmp/
+
+# Spark client shaded jar copied into the docker build context by build-image.sh
+docker/spark/*.jar


just exclude *.jar globally?

pan3793 · 2026-08-20T15:02:56Z

+BASE_SPARK_IMAGE_TAG="${CELEBORN_SPARK_BASE_TAG:-3.5.0}"
+
+echo "==> Building distribution (./build/make-distribution.sh $*) ..."
+./build/make-distribution.sh "$@"


we prefer to add --sbt-enabled

pan3793 · 2026-08-20T15:04:03Z

+# Only build it when make-distribution produced a Spark client shaded jar, which
+# happens when a -Pspark-3.x profile was passed. If absent, skip gracefully so
+# the Celeborn-only workflow keeps working.
+SPARK_CLIENT_JAR="$(ls dist/spark/celeborn-client-spark-3-shaded_*.jar 2>/dev/null || true)"


what about spark 4?

pan3793 · 2026-08-20T15:08:22Z

+docker exec celeborn-spark-master /opt/spark/bin/spark-submit \
+  --master spark://celeborn-spark-master:7077 \
+  --class org.apache.spark.examples.GroupByTest \
+  /opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar 10 100 1000 10


replace with the version-agnostic bin/run-example

Suggested change

docker exec celeborn-spark-master /opt/spark/bin/spark-submit \

--master spark://celeborn-spark-master:7077 \

--class org.apache.spark.examples.GroupByTest \

/opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar 10 100 1000 10

docker exec celeborn-spark-master /opt/spark/bin/run-example \

--master spark://celeborn-spark-master:7077 \

GroupByTest 10 100 1000 10

pan3793 · 2026-08-20T15:08:49Z

+# Usage (built by build-image.sh, not directly):
+#   docker build -t celeborn-spark:dev docker/spark/
+
+ARG spark_image_tag=3.5.0


Why choose 3.5.0 instead of 3.5.9?

Bumped to 3.5.9

- .gitignore: exclude *.jar globally instead of docker/spark/*.jar - build-image.sh: pass --sbt-enabled to make-distribution.sh - build-image.sh/Dockerfile: support Spark 4 client jar, derive default base image tag from Spark major version (3 -> 3.5.9, 4 -> 4.0.4) - Dockerfile: bump default apache/spark base tag from 3.5.0 to 3.5.9 - README.md: use version-agnostic bin/run-example in the example

github-actions Bot added kind:build kind:infra labels Jul 25, 2026

yew1eb force-pushed the add-docker-compose branch 3 times, most recently from e17f836 to 0c8a74e Compare July 25, 2026 07:27

github-actions Bot removed the kind:infra label Jul 25, 2026

[CELEBORN-2390] Add docker-compose for local cluster

b6fce04

yew1eb force-pushed the add-docker-compose branch from 0c8a74e to b6fce04 Compare July 25, 2026 07:33

yew1eb added 2 commits July 26, 2026 21:36

[CELEBORN-2390] Trigger CI

0e20a46

[CELEBORN-2390] Retrigger CI

c6375d1

yew1eb force-pushed the add-docker-compose branch from 116e390 to 4d37e7d Compare August 13, 2026 09:49

github-actions Bot added the kind:infra label Aug 13, 2026

pan3793 reviewed Aug 20, 2026

View reviewed changes

Uh oh!

Conversation

yew1eb commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR resolve a correctness bug?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

yew1eb commented Jul 31, 2026

Uh oh!

yew1eb commented Aug 4, 2026

Uh oh!

pan3793 commented Aug 13, 2026

Uh oh!

yew1eb commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yew1eb commented Jul 25, 2026 •

edited

Loading

yew1eb commented Aug 13, 2026 •

edited

Loading