docs(slurm): add public workflow and release checks

[andreatnvidia]

📋 Summary

Publish the complete public Fern workflow for Data Designer Slurm and tighten the package release checks that keep the optional extension installable, isolated, and consistent with the documented examples. This closes the documentation and release-integration slice without claiming the formal sealed-cluster acceptance owned by #870.

🔗 Related Issue

Closes #874

Formal sealed-artifact and real-cluster acceptance remains in #870. It follows after this PR merges and any required fixes land.

🔄 Changes

✨ Added

• Add the full Fern Slurm guide, including install, profile, image, execution, retry, collection, benchmark, dependency, troubleshooting, and CLI workflows.
• Add sanitized builder, run, benchmark, and profile examples that are loaded from the installed wheel during release validation.
• Add a legacy Slurm documentation coverage map with the Harden the Slurm runtime and validate sealed release artifacts #870 acceptance boundary recorded explicitly.

🔧 Changed

• Expand the public-artifact audit to cover the Slurm docs, navigation, examples, and coverage map.
• Strengthen wheel validation for all four same-version distributions, exact dependency edges, base-only isolation, lazy extension loading, offline wheelhouse installs, and a local PEP 503 index with ambient pip configuration disabled.
• Link the Slurm package README and Fern navigation to the new public material.

🐛 Fixed

• Remove the unsupported execute --force path from the CLI and public run-service API while preserving force handling where retry and benchmark workflows support it.
• Correct example YAML typing and document the shipped public-index and runtime-secret boundaries.
• Correct topology and benchmark guidance, document enforced array constraints, and remove internal milestone labels from public help and validation errors.
• Harden onboarding with Python preflight, site-safe setup guidance, explicit shared profiles and cluster selection, package verification, profile resource review, and compatible-image prerequisites.
• Clarify GPU request behavior, scheduler command paths, served-model configuration, timeout constraints, dry-run persistence, and winner-driven result discovery.
• Preserve allocation failure phase and exit status in batch stderr, with durable log paths for failed synchronous runtime steps.

🔍 Attention Areas

⚠️ Reviewers: Please pay special attention to the following:

• Release install validation - verifies wheel composition, dependency closure, isolation, examples, and CLI startup behavior.
• Public run service - removes the dead public force argument from execution.
• @nabinchha, please review legacy documentation parity, the operational workflow, and the sanitized public examples.

🧪 Testing

• make check-slurm
• make test-slurm - 1672 passed
• make test-slurm-wheel-install
• Hostile ambient pip configuration wheel-install test
• make check-fern-links
• make check-fern-docs - 0 errors, 1 pre-existing hidden-page warning
• .venv/bin/ruff check --fix . && .venv/bin/ruff format .
• Unit and integration tests added/updated
• Exact-wheel one-GPU documentation walkthrough completed successfully
• Formal sealed real-cluster acceptance remains tracked in Harden the Slurm runtime and validate sealed release artifacts #870

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO)
• Architecture docs updated (N/A - no architecture change)

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no actionable correctness, security, or repository-rule violations remain.

Summary

This PR publishes the public Slurm workflow and strengthens release validation for the optional extension.

• Adds navigable Fern documentation covering setup, configuration, execution, recovery, collection, benchmarking, dependencies, and troubleshooting.
• Adds packaged, sanitized examples and verifies that documentation embeds them verbatim.
• Expands wheel checks for aligned first-party versions, dependency isolation, offline installation, and local-index installation.
• Removes the unsupported execute-time force option while retaining force behavior for retry and benchmark workflows.
• Improves allocation and step failure diagnostics while preserving exit status and cleanup behavior.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Docs[Fern Slurm documentation] --> Examples[Packaged YAML examples]
  Examples --> Wheel[Data Designer Slurm wheel]
  Wheel --> Validation[Release-install validation]
  Validation --> BaseOnly[Base-only isolation]
  Validation --> Offline[Offline wheelhouse install]
  Validation --> Index[Local PEP 503 index install]
  Validation --> CLI[Lazy Slurm CLI startup]
  CLI --> Workflow[Profile → Image → Execute → Status → Retry → Merge]

Loading

_{Reviews (6) · Last reviewed commit: "docs(slurm): clarify verified onboarding..."}

[github-actions]

Fern preview: https://nvidia-preview-pr-939.docs.buildwithfern.com/nemo/datadesigner

Fern previews include the docs-website version archive with PR changes synced into latest. Notebook tutorials are rendered without execution outputs in previews.

[nabinchha]

Nice work on this one, @andreatnvidia — the public workflow and release hardening fit together cleanly.

Summary

This adds the full public Fern workflow for the Slurm extension, keeps the shipped examples synchronized with the guide, removes the unsupported force path from ordinary execution, and strengthens release validation for the optional wheel. The implementation matches the PR description and preserves force only for retry and benchmark recovery.

Findings

No blocking findings.

What Looks Good

• The docs are organized around the actual operator lifecycle, with cluster-specific facts kept in profiles and portable run intent kept in the run schema.
• The public examples are release-checked against the exact fenced snippets in Fern, and the expanded artifact audit covers the new docs, examples, navigation, and coverage map.
• The execution API cleanup is consistent across the CLI, service protocol, production wiring, benchmark orchestration, and tests. I also found no newly introduced avoidable nested functions.

Residual Risk

I could not independently complete the end-to-end wheel-install check because PyPI egress was unavailable in the isolated review environment. I reviewed that script in full and independently verified the Fern links, documented-example synchronization, public-artifact audit, formatting/lint, all 129 changed test slices, and 1,667 source-level Slurm tests; the remaining package-registration assertion requires installed wheel metadata.

Verdict

Ship it — ready to merge as-is.

---

This review was generated by an AI assistant.

[andreatnvidia]

Follow-up validation: I ran a clean-room documentation exercise with an agent that had no prior Data Designer Slurm implementation context. I provided only the cluster access instructions and operating rules, including using Slurm for compute, avoiding substantive work on the login node, and limiting the test to one GPU.

The agent followed the public workflow from environment and profile setup through image registration, dry-run, submission, status inspection, and result verification. The iterations produced the following material corrections:

• Added a Python 3.10+ preflight and clarified that setup must run in a site-approved submit environment, with substantive work kept off login nodes when required by site policy.
• Made the shared workspace, profile file, and cluster selection explicit across initialization, validation, image, execution, status, cancel, and merge commands.
• Added verification that the Slurm extension was actually installed, including guidance for releases or indexes that do not publish the slurm extra and its aligned wheel.
• Clarified that the image and model values are placeholders, that image construction is an operator prerequisite, and that client package versions, serving runtime, image visibility, and model visibility must be compatible before registration.
• Added review guidance for generated account, partition, GPU, workspace, and image-build resource defaults.
• Documented that gpus_per_node: auto can request every GPU reported by eligible nodes, and that the one-GPU walkthrough should use an explicit value of 1 when partial-node allocations are supported.
• Documented scheduler.bin_path for sites where Slurm commands are outside the generated batch job's default PATH.
• Corrected benchmark case terminology and replica-count guidance, multi-node tensor-parallel guidance, and the enforced limitations for array runs.
• Documented that served_model_name belongs on the deployment, that orchestrator-owned vLLM flags must not be placed in extra_args, and that distributed_init_timeout must not exceed startup_timeout.
• Clarified that dry-run staging is temporary and does not create resumable managed-run state.
• Documented that canonical results must be resolved through each shard's winner.json and referenced output-manifest.json, rather than by recursively counting data files from all attempts.
• Removed internal milestone and legacy product terminology from public help, validation messages, and the documentation coverage source.

The first runtime failure also showed that allocation-level errors did not provide enough durable context. 84fed0f6 now preserves the failing phase and exit status in batch stderr and reports the durable log paths for failed synchronous steps.

After those corrections, I rebuilt the packages from the exact branch commit, installed them into a fresh client image, and repeated the workflow. The one-GPU, one-record vLLM run completed successfully: the endpoint became ready, the attempt succeeded, the winner was published, and the canonical manifest identified exactly one output row containing the expected smoke marker.

The clean-room findings and resulting corrections span 570b7238, 6466007f, 84fed0f6, and 0f5a053d. The immediately preceding e09fe762 contains the related public-content sanitization. Together these provide additional end-to-end confidence in the public onboarding flow, but they are not a substitute for the formal sealed-cluster acceptance tracked by #870.