Skip to content

fix(deployment): validate launcher names instead of silently defaulting #174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
coketaste wants to merge 7 commits into
base: develop
Choose a base branch
from
Open

fix(deployment): validate launcher names instead of silently defaulting #174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
15be1c4
fix(examples): correct megatron-lm launcher value and dead master_por…
coketaste Aug 5, 2026
faca8d4
fix(deployment): validate launcher names instead of silently defaulting
coketaste Aug 20, 2026
d8d1aa1
Merge branch 'develop' into fix/launcher-validation
coketaste Sep 1, 2026
cacc4b7
fix(deployment): reject launcher sentinels at config boundaries
coketaste Sep 1, 2026
b275c07
Merge branch 'develop' into fix/launcher-validation
coketaste Sep 8, 2026
55bf9f1
Merge branch 'develop' into fix/launcher-validation
coketaste Sep 9, 2026
11c315e
chore: stop tracking superpowers plan/spec docs
coketaste Sep 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,9 @@ instance/
# Sphinx documentation
docs/_build/

# Local agent planning artifacts
docs/superpowers/

# Jupyter Notebook
.ipynb_checkpoints

Expand Down
2 changes: 1 addition & 1 deletion docs/deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ This creates:
**Supported Launchers:**
- `torchrun` - PyTorch DDP/FSDP
- `deepspeed` - ZeRO optimization
- `megatron` - Megatron-LM training
- `megatron-lm` - Megatron-LM training
- `torchtitan` - LLM pre-training
- `primus` - Primus unified pretrain (Megatron / TorchTitan / MaxText YAML)
- `vllm` - LLM inference
Expand Down
1,623 changes: 0 additions & 1,623 deletions docs/superpowers/plans/2026-08-27-pinned-image-digest.md
View file
Open in desktop

This file was deleted.

156 changes: 0 additions & 156 deletions docs/superpowers/specs/2026-08-27-pinned-image-digest-design.md
View file
Open in desktop

This file was deleted.

6 changes: 3 additions & 3 deletions examples/k8s-configs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ To validate rendered YAML after a debug run, install [kubeconform](https://githu

### Multi-node DNS (PyTorch vs Ray)

For **PyTorch-native** launchers (`torchrun`, `deepspeed`, `torchtitan`, `megatron`, `primus`), multi-node Jobs use a **headless Service** whose name matches `pod.spec.subdomain`, per Kubernetes DNS rules, so pods get stable per-pod DNS names for rendezvous.
For **PyTorch-native** launchers (`torchrun`, `deepspeed`, `torchtitan`, `megatron-lm`, `primus`), multi-node Jobs use a **headless Service** whose name matches `pod.spec.subdomain`, per Kubernetes DNS rules, so pods get stable per-pod DNS names for rendezvous.

For **Ray-based** multi-node (`vllm`, `sglang`), a headless Service may still be created for networking, but **per-pod DNS via `subdomain` is not applied** the same way as for PyTorch; production multi-node Ray on Kubernetes often uses **KubeRay** (see upstream vLLM / Ray docs). Treat Job-based multi-node Ray as a best-effort path.

Expand Down Expand Up @@ -580,7 +580,7 @@ Configuration for distributed workloads (training and inference):

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `launcher` | string | - | Launcher type: `torchrun`, `deepspeed`, `torchtitan`, `megatron`, `primus`, `vllm`, `sglang` |
| `launcher` | string | - | Launcher type: `torchrun`, `deepspeed`, `torchtitan`, `megatron-lm`, `primus`, `vllm`, `sglang` |
| `enabled` | boolean | `false` | Enable distributed execution (legacy, prefer `launcher`) |
| `backend` | string | `"nccl"` | `"nccl"`, `"gloo"`, or `"mpi"` |
| `nnodes` | integer | `1` | Number of nodes |
Expand Down Expand Up @@ -679,7 +679,7 @@ Write durable outputs under `/results/<replica-id>/` in the container so each re
**Training Launchers:**
- **torchrun**: Standard PyTorch DDP/FSDP training
- **deepspeed**: ZeRO optimization for memory efficiency
- **megatron**: Megatron-LM tensor and pipeline parallelism
- **megatron-lm**: Megatron-LM tensor and pipeline parallelism
- **torchtitan**: LLM pre-training with multi-dimensional parallelism (FSDP2+TP+PP)
- **primus**: Unified Primus pretrain (Megatron / TorchTitan / MaxText experiment YAML; see [Primus on Kubernetes](#primus-on-kubernetes))

Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu-tools.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"launcher": "torchrun",
"nnodes": 1,
"nproc_per_node": 2,
"master_port": 29500
"port": 29500
},

"env_vars": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/02-torchrun-single-node-multi-gpu.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"launcher": "torchrun",
"nnodes": 1,
"nproc_per_node": 2,
"master_port": 29500
"port": 29500
},

"env_vars": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/03-torchrun-multi-node-basic.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"launcher": "torchrun",
"nnodes": 2,
"nproc_per_node": 2,
"master_port": 29500
"port": 29500
},

"env_vars": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/04-torchrun-multi-node-advanced.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
"launcher": "torchrun",
"nnodes": 4,
"nproc_per_node": 2,
"master_port": 29500
"port": 29500
},

"env_vars": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/05-torchrun-nvidia-gpu-example.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
"launcher": "torchrun",
"nnodes": 1,
"nproc_per_node": 4,
"master_port": 29500
"port": 29500
},

"env_vars": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/06-data-provider-with-pvc.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"nnodes": 1,
"nproc_per_node": 2,

"master_port": 29500
"port": 29500
},

"env_vars": {
Expand Down
4 changes: 2 additions & 2 deletions examples/k8s-configs/basic/megatron-lm-multi-node-basic.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@
},

"distributed": {
"launcher": "megatron",
"launcher": "megatron-lm",
"nnodes": 4,
"nproc_per_node": 8,
"master_port": 29500
"port": 29500
},

"env_vars": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/sglang-disagg-custom-split.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"launcher": "sglang-disagg",
"nnodes": 7,
"nproc_per_node": 8,
"master_port": 29500,
"port": 29500,
"sglang_disagg": {
"prefill_nodes": 4,
"decode_nodes": 2
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/sglang-disagg-multi-node-basic.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"launcher": "sglang-disagg",
"nnodes": 5,
"nproc_per_node": 8,
"master_port": 29500
"port": 29500
},

"context": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/sglang-multi-node-basic.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"launcher": "sglang",
"nnodes": 2,
"nproc_per_node": 4,
"master_port": 29500
"port": 29500
},

"context": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/torchtitan-multi-node-basic.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"launcher": "torchtitan",
"nnodes": 4,
"nproc_per_node": 8,
"master_port": 29500
"port": 29500
},

"context": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/basic/vllm-multi-node-basic.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"launcher": "vllm",
"nnodes": 2,
"nproc_per_node": 4,
"master_port": 29500
"port": 29500
},

"context": {
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/minimal/megatron-lm-exclude-node.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
},

"distributed": {
"launcher": "megatron",
"launcher": "megatron-lm",
"nnodes": 1,
"nproc_per_node": 2
},
Expand Down
2 changes: 1 addition & 1 deletion examples/k8s-configs/minimal/megatron-lm-minimal.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
},

"distributed": {
"launcher": "megatron",
"launcher": "megatron-lm",
"nnodes": 1,
"nproc_per_node": 2
},
Expand Down
4 changes: 2 additions & 2 deletions examples/k8s-configs/minimal/megatron-lm-optimized.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,10 @@
"distributed": {
"enabled": true,
"backend": "nccl",
"launcher": "megatron",
"launcher": "megatron-lm",
"nnodes": 1,
"nproc_per_node": 2,
"master_port": 29500
"port": 29500
},

"env_vars": {
Expand Down
6 changes: 3 additions & 3 deletions examples/slurm-configs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ madengine uses intelligent multi-layer configuration merging:
```json
{
"distributed": {
"launcher": "torchrun", // Launcher type: torchrun, vllm, sglang, deepspeed, megatron, slurm_multi
"launcher": "torchrun", // Launcher type: torchrun, vllm, sglang, deepspeed, megatron-lm, slurm_multi
"backend": "nccl", // Communication backend (nccl/gloo)
"port": 29500, // Master node port
"nnodes": 2, // Number of nodes (overrides slurm.nodes if set)
Expand All @@ -426,12 +426,12 @@ madengine uses intelligent multi-layer configuration merging:
- `vllm`: vLLM inference engine (TP/PP parallelism)
- `sglang`: SGLang inference engine
- `deepspeed`: DeepSpeed training framework
- `megatron`: Megatron-LM large model training
- `megatron-lm`: Megatron-LM large model training
- `slurm_multi` / `slurm-multi`: Self-managed multi-container topologies (escape hatch)
- Custom: Set environment variables, model script handles launcher

**Note**: For vLLM and SGLang, the model script handles process spawning directly.
For torchrun/deepspeed/megatron, use `$MAD_MULTI_NODE_RUNNER` in your model script.
For torchrun/deepspeed/megatron-lm, use `$MAD_MULTI_NODE_RUNNER` in your model script.
For slurm_multi, the model's `.slurm` script runs on baremetal and manages Docker containers via `srun` internally.

### Environment Variables
Expand Down
4 changes: 2 additions & 2 deletions examples/slurm-configs/basic/09-megatron-lm-multi-node.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@
},

"distributed": {
"launcher": "megatron",
"launcher": "megatron-lm",
"nnodes": 4,
"nproc_per_node": 8,
"master_port": 29500
"port": 29500
},

"env_vars": {
Expand Down
2 changes: 1 addition & 1 deletion examples/slurm-configs/minimal/megatron-lm-minimal.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
},

"distributed": {
"launcher": "megatron",
"launcher": "megatron-lm",
"nnodes": 1,
"nproc_per_node": 2
},
Expand Down
Loading