fix: create the external metrics tracker only on trainer rank zero - #782
Merged
jiapingW merged 3 commits intoSep 2, 2026
Merged
Conversation
5 tasks
This was referenced Sep 1, 2026
maocheng23
force-pushed
the
maocheng/colocate-2-teardown-abort
branch
from
September 1, 2026 23:30
b8f2ba5 to
6a5cccd
Compare
maocheng23
force-pushed
the
maocheng/colocate-3-rank0-tracker
branch
from
September 1, 2026 23:30
48a2e3e to
fb2bd50
Compare
Expose capture_rows(input_ids) on OfflineSGLangCaptureBackend and OfflineSGLangCapture so callers can capture variable-length rows in one packed prefill without building padded tensors. capture_eagle3 delegates to it and keeps its exact request construction and outputs. DSpark capture-layer setup now falls back from the native set_dspark_layers_to_capture hook to the dense DFlash hook (with a log line naming the resolved hook) so dense targets such as Qwen3 can serve DSpark capture on stock SGLang 0.5.14 models. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every trainer rank constructed its own tracker, so W&B/MLflow runs were duplicated world-size times and TensorBoard ranks wrote the same directory. An external tracker is now created on global rank zero only; every rank keeps the console logger, so console output is unchanged.
maocheng23
force-pushed
the
maocheng/colocate-2-teardown-abort
branch
from
September 2, 2026 00:24
6a5cccd to
bb22955
Compare
maocheng23
force-pushed
the
maocheng/colocate-3-rank0-tracker
branch
from
September 2, 2026 00:24
fb2bd50 to
9edfa1b
Compare
maocheng23
marked this pull request as ready for review
September 2, 2026 00:25
…rows refactor: extract packed capture_rows from the offline SGLang backend
jiapingW
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Stack 2/5 replacing draft #766. Every trainer rank currently constructs its own tracker: W&B/MLflow runs are duplicated world-size times and TensorBoard ranks write the same directory. A 32-rank colocated K3 job would create 32 W&B runs; existing multi-rank disaggregated consumers (the K3 disaggregated recipe runs 4) already suffer this.
Modifications
_configured_loggercreates the external tracker (W&B, MLflow, SwanLab, TensorBoard) on global rank zero only. Every rank keeps the console logger, so console output on multi-rank runs is unchanged; only the duplicate tracker runs/writers go away. Producers are single-process roles and are unaffected.Related Issues
Splits #766. Stack: #781 teardown-abort ← #2 (this) ← #780 capture-rows ← #783 colocated-core ← #784 hybrid-shard.
Accuracy Test
test_only_global_rank_zero_creates_an_external_tracker(rank 3 withreport_to: wandbgets the console logger and no tracker is created) andtest_primary_rank_keeps_the_console_logger_without_a_tracker; the existing tracker-adapter test covers rank 0 creating the tracker.Checklist
black --checkandisort --check-only).