Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions fern/versions/latest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,28 @@ navigation:
path: ./latest/pages/notebooks/6-editing-images-with-image-context.mdx
- section: Slurm
contents:
- page: Overview
path: ./latest/pages/slurm/overview.mdx
- page: Getting Started
path: ./latest/pages/slurm/getting-started.mdx
- page: Profiles and Clusters
path: ./latest/pages/slurm/profiles.mdx
- page: Images
path: ./latest/pages/slurm/images.mdx
- page: Run Configuration
path: ./latest/pages/slurm/run-configuration.mdx
- page: Run Operations
path: ./latest/pages/slurm/operations.mdx
- page: Retry and Collection
path: ./latest/pages/slurm/retry-and-collection.mdx
- page: Benchmarks
path: ./latest/pages/slurm/benchmarks.mdx
- page: Dependencies and Tools
path: ./latest/pages/slurm/dependencies-and-tools.mdx
- page: Troubleshooting
path: ./latest/pages/slurm/troubleshooting.mdx
- page: CLI Reference
path: ./latest/pages/slurm/cli-reference.mdx
- section: Recipes
contents:
- page: Recipe Cards
Expand Down
12 changes: 9 additions & 3 deletions fern/versions/latest/pages/slurm/benchmarks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Use a benchmark to compare concurrency and deployment topology while keeping each case as an ordinary Data Designer Slurm run. No benchmark controller stays resident after submission.

Complete [Getting Started](/slurm/getting-started) first so the profile, images, and base run are ready.

Install the Slurm extension:

```bash
Expand All @@ -22,12 +24,12 @@ concurrency_values:
- 32
- 64
deployment_cases:
- name: two-independent-replicas
- name: single-node-groups
deployments:
generator:
nodes: 2
nodes_per_replica: 1
- name: one-two-node-replica
- name: two-node-group
deployments:
generator:
nodes: 2
Expand All @@ -42,7 +44,9 @@ analysis:
target_runtime: 4h
```

The compiler expands the authored order deterministically. For this example, the first cases are `two-independent-replicas-c32` and `one-two-node-replica-c32`. An adaptive record policy uses `ceil(concurrency * records_per_concurrency)`, bounded by `base_records` and `max_records`.
The compiler expands the authored order deterministically. For this example, the first cases are `single-node-groups-c32` and `two-node-group-c32`. An adaptive record policy uses `ceil(concurrency * records_per_concurrency)`, bounded by `base_records` and `max_records`.

`nodes_per_replica` controls the number of nodes in each pipeline-parallel group. The resolved replica count is `(nodes / nodes_per_replica) * (gpus_per_node / tensor_parallel)`, so it also depends on the selected profile and the base run's deployment topology.

## Run

Expand Down Expand Up @@ -92,3 +96,5 @@ service = create_slurm_benchmark_service(
manifest = service.run(config, source_root=benchmark_file.parent)
report = service.analyze(manifest.benchmark_id, refresh_state=True)
```

For child failure recovery and output collection, use [Retry and collection](/slurm/retry-and-collection).
42 changes: 42 additions & 0 deletions fern/versions/latest/pages/slurm/cli-reference.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Slurm CLI reference

Install the extension with `python -m pip install "data-designer[slurm]"`, verify it with `python -m pip show data-designer-slurm`, then use `data-designer slurm COMMAND --help` for the authoritative option list.

All commands accept `--profile-file PATH` and `--cluster NAME` where cluster selection is required.

| Command | Purpose |
| --- | --- |
| `profile init` | Create a restrictive starter profile catalog without overwriting. |
| `profile validate` | Validate catalog selection, shared workspace access, and scheduler facts. |
| `image add` | Import or inspect an image through a CPU Slurm job and publish an alias. |
| `image ls` | List aliases in the selected profile workspace. |
| `image info NAME` | Show one registered alias and immutable image facts. |
| `image rm NAME` | Remove an alias without deleting its SQSH artifact. |
| `execute RUN_FILE` | Resolve and submit a run. Add `--dry-run` to render without submission. |
| `status RUN_ID` | Reconcile scheduler observations with durable managed state. |
| `cancel RUN_ID` | Cancel active jobs owned by a managed run. |
| `retry RUN_OR_JOB_ID` | Preview or submit a sparse retry. Repeat `--task-id` to select shards. |
| `merge` | Submit a zero-GPU winner collection job for a managed run directory. |
| `benchmark run BENCHMARK_FILE` | Expand benchmark cases into ordinary child runs and submit them. |
| `benchmark analyze BENCHMARK` | Write a report from persisted child state. |

## Common options

| Option | Commands | Meaning |
| --- | --- | --- |
| `--profile-file PATH` | Most commands | Use a catalog other than the default or environment-selected file. |
| `--cluster NAME` | Most commands | Select one catalog entry explicitly. |
| `--dry-run` | `execute`, `retry` | Perform resolution and render a batch script without submission. |
| `--force` | `retry`, `benchmark run` | Skip retry confirmation or resume a safely recoverable benchmark submission. It does not overwrite immutable evidence. |
| `--refresh-state` | `benchmark analyze` | Reconcile each child with the scheduler before analysis. |
| `--fail-if-incomplete` | `benchmark analyze` | Persist the report, then return a conflict if any case is incomplete. |

## Output and errors

Success is one canonical JSON value on stdout. Failure is one object on stderr:

```json
{"error":{"code":"invalid_request","message":"...","operation":"execute"}}
```

Exit codes are `1` for internal errors, `2` for invalid requests, `3` for missing managed resources, `4` for conflicts, and `5` for unavailable external systems. Error messages are bounded and sanitized; inspect durable logs for operational detail.
115 changes: 115 additions & 0 deletions fern/versions/latest/pages/slurm/dependencies-and-tools.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Dependencies, plugins, MCP, and secrets

The client image provides Data Designer and its base environment. A run may add immutable pure Python wheels as an allocation-local overlay. The overlay is reconstructed for each allocation and is not written back into the image.

## Add package requirements

```yaml
client:
image:
name: dd-client
dependencies:
requirements:
- example-plugin==1.2.3
```

The submit-side resolver accepts package requirements or direct HTTPS wheel URLs with a SHA-256 fragment. It rejects editable installs, source directories, VCS references, markers, duplicate normalized names, and non-wheel URLs. The resolved lock and wheels are persisted with the run.

Do not use the overlay to replace `data-designer`, `data-designer-config`, `data-designer-engine`, or `data-designer-slurm`. Build a new client image when these packages need to change.

## Reuse a dependency lock

Use exactly one of `requirements` or `lock_file`:

```yaml
client:
image:
name: dd-client
dependencies:
requirements: null
lock_file: inputs/dependency-lock.json
```

The lock must target the registered client image inventory and reference available wheel artifacts.

`client.dependencies.index_credentials` reserves named secret references for the allocation runtime. Submit-side inline requirement resolution does not use them and always targets the public index.

Inline requirements resolve from the public Python package index with ambient pip configuration disabled. For private or unpublished plugins, bake the package into the client image, use an immutable HTTPS wheel URL that does not embed credentials, or supply a previously resolved compatible lock and wheel artifacts.

## Reference secrets

Secrets are environment references, never literal values in persisted YAML:

```yaml
deployments:
- model_alias: generator
model: example/generator
server:
type: vllm
image:
name: vllm
environment:
MODEL_ACCESS_TOKEN:
type: secret
environment: MODEL_ACCESS_TOKEN
```

Literal non-secret settings use an explicit literal binding:

```yaml
environment:
VLLM_LOGGING_LEVEL:
type: literal
value: INFO
MODEL_ACCESS_TOKEN:
type: secret
environment: MODEL_ACCESS_TOKEN
```

Export referenced variables in the trusted submit environment before running `execute`. Values are forwarded only at the required runtime boundary and are not serialized into run contracts or logs. Secret-shaped environment names cannot use literal bindings.

## Use plugins

Install plugin packages into the client image or declare a compatible pure-wheel dependency. The serialized builder config can then reference the plugin column, seed reader, or processor. See the [plugin overview](/plugins/overview) for Data Designer plugin contracts.

## Configure MCP providers

Remote providers support SSE and streamable HTTP:

```yaml
invocation:
num_records: 100
dataset_name: tool-use
mcp_providers:
- provider_type: streamable_http
name: search
endpoint: https://tools.example.test/mcp
api_key:
type: secret
environment: MCP_API_KEY
```

Endpoints cannot contain credentials, query parameters, or fragments.

Local stdio providers run inside the client allocation:

```yaml
invocation:
num_records: 100
dataset_name: tool-use
mcp_providers:
- provider_type: stdio
name: local-tool
command: example-mcp-server
args:
- --mode
- read-only
environment:
TOOL_API_KEY:
type: secret
environment: TOOL_API_KEY
```

The command is one executable token and arguments are passed without a shell. The client image or dependency overlay must provide the executable. See [Tool Use and MCP](/concepts/tool-use-mcp/overview) for builder-side tool configuration.

Code-sandbox sidecars are not supported by the v1 Slurm extension. Use an MCP service with an appropriate isolation boundary or run the workload through a separately managed sandbox system.
Loading
Loading