Skip to content

fix: create the external metrics tracker only on trainer rank zero - #782

Merged
jiapingW merged 3 commits into
maocheng/colocate-2-teardown-abortfrom
maocheng/colocate-3-rank0-tracker
Sep 2, 2026
Merged

fix: create the external metrics tracker only on trainer rank zero#782
jiapingW merged 3 commits into
maocheng/colocate-2-teardown-abortfrom
maocheng/colocate-3-rank0-tracker

Conversation

@maocheng23

@maocheng23 maocheng23 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Motivation

Stack 2/5 replacing draft #766. Every trainer rank currently constructs its own tracker: W&B/MLflow runs are duplicated world-size times and TensorBoard ranks write the same directory. A 32-rank colocated K3 job would create 32 W&B runs; existing multi-rank disaggregated consumers (the K3 disaggregated recipe runs 4) already suffer this.

Modifications

  • _configured_logger creates the external tracker (W&B, MLflow, SwanLab, TensorBoard) on global rank zero only. Every rank keeps the console logger, so console output on multi-rank runs is unchanged; only the duplicate tracker runs/writers go away. Producers are single-process roles and are unaffected.

Related Issues

Splits #766. Stack: #781 teardown-abort ← #2 (this)#780 capture-rows ← #783 colocated-core ← #784 hybrid-shard.

Accuracy Test

  • test_only_global_rank_zero_creates_an_external_tracker (rank 3 with report_to: wandb gets the console logger and no tracker is created) and test_primary_rank_keeps_the_console_logger_without_a_tracker; the existing tracker-adapter test covers rank 0 creating the tracker.

Checklist

  • Format your code according to the Code Formatting with Pre-Commit (black --check and isort --check-only).
  • Add unit tests.
  • Update documentation as needed (none needed).

maocheng23 and others added 2 commits September 1, 2026 17:23
Expose capture_rows(input_ids) on OfflineSGLangCaptureBackend and
OfflineSGLangCapture so callers can capture variable-length rows in one
packed prefill without building padded tensors. capture_eagle3 delegates
to it and keeps its exact request construction and outputs.

DSpark capture-layer setup now falls back from the native
set_dspark_layers_to_capture hook to the dense DFlash hook (with a log
line naming the resolved hook) so dense targets such as Qwen3 can serve
DSpark capture on stock SGLang 0.5.14 models.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every trainer rank constructed its own tracker, so W&B/MLflow runs were
duplicated world-size times and TensorBoard ranks wrote the same directory.
An external tracker is now created on global rank zero only; every rank keeps
the console logger, so console output is unchanged.
@maocheng23 maocheng23 changed the title fix: create the metrics tracker only on trainer rank zero fix: create the external metrics tracker only on trainer rank zero Sep 2, 2026
@maocheng23
maocheng23 force-pushed the maocheng/colocate-2-teardown-abort branch from 6a5cccd to bb22955 Compare September 2, 2026 00:24
@maocheng23
maocheng23 force-pushed the maocheng/colocate-3-rank0-tracker branch from fb2bd50 to 9edfa1b Compare September 2, 2026 00:24
@maocheng23
maocheng23 marked this pull request as ready for review September 2, 2026 00:25
…rows

refactor: extract packed capture_rows from the offline SGLang backend
@jiapingW
jiapingW merged commit dfc2f36 into maocheng/colocate-2-teardown-abort Sep 2, 2026
1 check passed
@jiapingW
jiapingW deleted the maocheng/colocate-3-rank0-tracker branch September 2, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants