Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
f1618bf
added spack source mirrors capability, needs testing
grodzki-lanl Feb 17, 2026
954a690
removed lanl stuff
grodzki-lanl Feb 18, 2026
97d43a0
add source mirrors via config.yaml
Feb 24, 2026
15e97f1
add source mirrors via config.yaml
Feb 24, 2026
d125185
Merge branch 'main' of https://github.com/grodzki-lanl/stackinator
Feb 24, 2026
f91fbad
add source mirrors via config.yaml and retain spack default mirror
grodzki-lanl Feb 24, 2026
2a6571e
fixed spaces/tabs typo
grodzki-lanl Feb 24, 2026
06f9dac
Added mirror configuration json schema.
Paul-Ferrell Mar 6, 2026
7692677
Incorporating Makefile changes.
Paul-Ferrell Mar 6, 2026
940886b
mirrors
grodzki-lanl Mar 6, 2026
ffb6083
mirrors
grodzki-lanl Mar 6, 2026
04f9908
validate mirror config
grodzki-lanl Mar 6, 2026
b45039d
Updating recipe to handle new mirrors format.
Paul-Ferrell Mar 6, 2026
d3ee8e4
Updating mirror configuration more.
Paul-Ferrell Mar 6, 2026
0ed23d8
mirror yaml generator
grodzki-lanl Mar 6, 2026
e2a67ab
update mirrors
grodzki-lanl Mar 6, 2026
7346386
validate keys in mirror config and fixed yaml generator
grodzki-lanl Mar 7, 2026
e722f49
validate keys in mirror config and fixed yaml generator
grodzki-lanl Mar 7, 2026
eb2685f
connecting mirrors to builder.py
grodzki-lanl Mar 9, 2026
9d40a83
Put the mirror manipulation code in a class.
Paul-Ferrell Mar 12, 2026
2a7bd1d
preserve cache for makefile
grodzki-lanl Mar 12, 2026
8589cff
Adding bootstrap mirror configs.
Paul-Ferrell Mar 12, 2026
d6a3df7
Reverted to defining the key store path in builder.
Paul-Ferrell Mar 12, 2026
ee0d45f
Compressed mirror config setup into a single interface.
Paul-Ferrell Mar 12, 2026
90d7733
Catching builder exceptions.
Paul-Ferrell Mar 12, 2026
7362cbc
fixing key setup
grodzki-lanl Mar 12, 2026
2a4632d
In progress.
Paul-Ferrell Mar 12, 2026
3915036
Enforce gcc~builtins (#284)
albestro Feb 26, 2026
e11685c
added GPG key verification
grodzki-lanl Mar 13, 2026
593159f
unit tests for mirrors
grodzki-lanl Mar 13, 2026
e1a40fd
mirrors.yaml is now a name:{<config>} mapping
Paul-Ferrell Mar 13, 2026
fb3289d
Now add mount specific paths to certain mirrors.
Paul-Ferrell Mar 13, 2026
d47e2f3
updated test mirror format
grodzki-lanl Mar 13, 2026
c8fb6f9
Fixed build cache enabling via cmdline.
Paul-Ferrell Mar 13, 2026
951aeb4
Error handling
Paul-Ferrell Mar 13, 2026
2ea49e9
Adding a unittest.
Paul-Ferrell Mar 13, 2026
c668b5b
updated yaml formatting
grodzki-lanl Mar 13, 2026
45b1331
Fixed error message.
Paul-Ferrell Mar 13, 2026
533a606
debugging
grodzki-lanl Mar 13, 2026
34d0c65
Minor fix.
Paul-Ferrell Mar 13, 2026
f989631
Fixing urls.
Paul-Ferrell Mar 13, 2026
7822fb7
fixed url for testing
grodzki-lanl Mar 13, 2026
8c814aa
validated mirror tests
grodzki-lanl Mar 13, 2026
5ab3279
added test description
grodzki-lanl Mar 13, 2026
5999ba4
Adding unit tests.
Paul-Ferrell Mar 16, 2026
3c7bc0a
Unit test tweaking.
Paul-Ferrell Mar 16, 2026
e19b868
Added unittests for key setup
Paul-Ferrell Mar 16, 2026
86047ca
Added test for bad keys.
Paul-Ferrell Mar 16, 2026
3d840d0
added more mirror tests
grodzki-lanl Mar 16, 2026
0a25459
added test for bad urls
grodzki-lanl Mar 16, 2026
35591bc
added requirements.txt
grodzki-lanl Mar 16, 2026
431cc9d
Fixed unittest.
Paul-Ferrell Mar 16, 2026
0c59ee9
Added one more unittest.
Paul-Ferrell Mar 16, 2026
d9cdfc4
Linting
Paul-Ferrell Mar 16, 2026
30521d3
Got rid of cache.py
Paul-Ferrell Mar 16, 2026
191af6f
use pyproject for dependencies
bcumming Mar 17, 2026
2fc29d6
add unit test and lint hints to readme
bcumming Mar 17, 2026
d8594a3
tweak presentation of build cache by recipe
bcumming Mar 17, 2026
ef9d7e8
fix build cache configuration
bcumming Mar 20, 2026
8b1e0af
modified bootstrap yaml setup and removed hardcoded alpscache
grodzki-lanl Mar 20, 2026
3c21789
mirrors documentation
grodzki-lanl Mar 26, 2026
596f50e
Switched to the requests libs from urlopen, as urlopen doesn't handle…
Paul-Ferrell Apr 1, 2026
ff42460
restrict number of buildcaches and bootstraps in mirror schema
grodzki-lanl Apr 8, 2026
e10f0c9
fixed mistake
grodzki-lanl Apr 8, 2026
6a203c3
refactored mirrors.py to match new schema
grodzki-lanl Apr 14, 2026
099878a
fixing unit tests, _key_init() still broken
grodzki-lanl Apr 21, 2026
17f4bc6
reverted to old key_setup, add handling for private keys, refactored …
grodzki-lanl Apr 23, 2026
4206e68
linting
grodzki-lanl Apr 23, 2026
fa7ef2f
added 'requests' to pyproject.toml
grodzki-lanl Apr 23, 2026
41e7630
removed old reference
grodzki-lanl Apr 27, 2026
daa8cff
sync with upstream
grodzki-lanl May 19, 2026
7f6b560
fixed documentation and cleaned up files
grodzki-lanl May 19, 2026
59901c3
ensure yaml matches dict order
grodzki-lanl May 19, 2026
8e575b0
linting
grodzki-lanl May 19, 2026
42502b4
wrap buildcache keys --install --trust with conditional
grodzki-lanl May 20, 2026
0a3bcff
wrong variable
grodzki-lanl May 20, 2026
ed05e2c
accept armored gpg keys
grodzki-lanl May 20, 2026
4ce0564
optimise url validation
bcumming Jun 4, 2026
cfa74e2
fix mount_specific behavior; remove "enabled" toggle
bcumming Jun 4, 2026
c60fc53
refactor mirror configuration
bcumming Jun 4, 2026
9c86155
add source caching
bcumming Jun 5, 2026
4a1c01c
support read only buildcaches; set defaults on mirrors.yaml inputs
bcumming Jun 5, 2026
38b0a7d
make mirrors a cli argument, not part of system config
bcumming Jun 5, 2026
3ebb484
support misc caches in mirrors: useful for concretization caches
bcumming Jun 5, 2026
5e88905
update agent instructions
bcumming Jun 5, 2026
d8fa707
tests from last commit
bcumming Jun 5, 2026
7c92da2
boostrap cache is read only without a key: simplify if it is on a loc…
bcumming Jun 5, 2026
adc0819
uv only checks for upates when necessary
bcumming Jun 10, 2026
6c265e1
do not re-index build cache every time we push
bcumming Jun 10, 2026
b8f847d
remove redundant key from source mirrors
bcumming Jun 10, 2026
840d85a
misc cache -> explicit concretizer cache
bcumming Jun 10, 2026
107ebc8
tidy up docs a wee bit
bcumming Jun 11, 2026
e1ed765
fix doc index
bcumming Jun 11, 2026
71e4812
use spack schema for mirrors
bcumming Jun 15, 2026
7ae87fd
update docs
bcumming Jun 16, 2026
7a34ac5
always generate push/fetch pairs - even for read only targets
bcumming Jun 16, 2026
c8dac09
clean up linking in docs
bcumming Jun 16, 2026
b201f3c
tweak footnotes
bcumming Jun 16, 2026
b99d3cc
improve warning for old --cache flag
bcumming Jun 16, 2026
b62f24d
fix unit tests
bcumming Jun 17, 2026
3e6efc1
merge main
bcumming Jun 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
350 changes: 350 additions & 0 deletions CLAUDE.md

Large diffs are not rendered by default.

23 changes: 23 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,26 @@ A tool for building a scientific software stack from a recipe for vClusters on C
Read the [documentation](https://eth-cscs.github.io/stackinator/) to get started.

Create a ticket in our [GitHub issues](https://github.com/eth-cscs/stackinator/issues) if you find a bug, have a feature request or have a question.

## running tests:

Use uv to run the tests, which will in turn ensure that the correct dependencies from `pyproject.toml` are used:

```
uv run pytest
```

Before pushing, apply the linting rules (this calls uv under the hood):

```
./lint
```
## building the docs

The documentation for Stackinator is built from the markdown files in the `docs` path using MkDocs and MkDocs-material.
You can view the latest documentation online at [github.io](https://eth-cscs.github.io/stackinator/)

To view work in progress docs, run the serve script and follow the link it provides to view a local copy of the docs in your browser.
```bash
./serve
```
3 changes: 2 additions & 1 deletion bin/stack-config
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
#!/usr/bin/env -S uv run --script
#!/usr/bin/env -S uv run --no-refresh --script
# /// script
# requires-python = ">=3.12"
# dependencies = [
# "python-magic",
# "jinja2",
# "jsonschema",
# "pyYAML",
Expand Down
101 changes: 0 additions & 101 deletions docs/build-caches.md

This file was deleted.

5 changes: 3 additions & 2 deletions docs/building.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[](){#ref-building}
# Building Spack Stacks

Once a stack has been [configured](configuring.md) using `stack-config`, it's time to build the software stack.
Once a stack has been [configured][ref-configuring] using `stack-config`, it's time to build the software stack.

## How to Build

Expand All @@ -18,7 +19,7 @@ env --ignore-environment PATH=/usr/bin:/bin:`pwd -P`/spack/bin make modules stor
The call to `make` is wrapped with with `env --ignore-env` to unset all environment variables, to improve reproducability of builds.

Build times for stacks typically vary between 30 minutes to 3 hours, depending on the specific packages that have to be built.
Using [build caches](build-caches.md) and building in shared memory (see below) are the most effective methods to speed up builds.
Using [build caches][ref-mirrors] and building in shared memory (see below) are the most effective methods to speed up builds.

## Where to Build

Expand Down
10 changes: 9 additions & 1 deletion docs/cluster-config.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
[](){#ref-cluster-config}
# Cluster Configuration

Spack stacks are built on bare-metal clusters using a minimum of dependencies from the underlying system.
Expand All @@ -10,7 +11,7 @@ A cluster configuration is a directory with the following structure:
└─ repos.yaml # optional reference to additional site packages
```

The configuration is provided during the [configuration](configuring.md) step with the `--system/-s` flag.
The configuration is provided during the [configuration][ref-configuring] step with the `--system/-s` flag.
The following example targets the Clariden system at CSCS:

```bash
Expand Down Expand Up @@ -93,6 +94,13 @@ packages:
version: ["git.59b6de6a91d9637809677c50cc48b607a91a9acb=main"]
```

### Configuring Spack mirrors

Mirrors and caches are **not** part of the system configuration.
They are supplied per-invocation with `stack-config --mirror <file>`, because the locations involved (build caches, source caches) are often specific to the user running the build and may not be accessible to everyone using a system.

See [Mirrors and Build Caches][ref-mirrors] for the full reference and examples.

## Site and System Configurations

The `repo.yaml` configuration can be used to provide a list of additional Spack package repositories to use on the target system.
Expand Down
10 changes: 6 additions & 4 deletions docs/configuring.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
[](){#ref-configuring}
# Configuring Spack Stacks

Stackinator generates the make files and spack configurations that build the spack environments that are packaged together in the spack stack.
Expand All @@ -10,13 +11,14 @@ stack-config --build $BUILD_PATH --recipe $RECIPE_PATH --system $SYSTEM_CONFIG_P

The following flags are required:

* `-b/--build`: the path where the [build](building.md) is to be performed.
* `-r/--recipe`: the path with the [recipe](recipes.md) yaml files that describe the environment.
* `-s/--system`: the path containing the [system configuration](cluster-config.md) for the target cluster.
* `-b/--build`: the path where the [build][ref-building] is to be performed.
* `-r/--recipe`: the path with the [recipe][ref-recipes] yaml files that describe the environment.
* `-s/--system`: the path containing the [system configuration][ref-cluster-config] for the target cluster.

The following flags are optional:

* `-c/--cache`: configure the [build cache](build-caches.md).
* `--mirror`: path to a [mirrors.yaml][ref-mirrors] file configuring build caches and mirrors.
* `-c/--cache`: legacy build cache configuration file (deprecated; use `--mirror`).
* `-m/--mount`: override the [mount point](installing.md) where the stack will be installed.
* `--version`: print the stackinator version.
* `-h/--help`: print help message.
Expand Down
5 changes: 4 additions & 1 deletion docs/installing.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
[](){#ref-installing}
# Installing Stacks

The installation path of the software stack is set when the stack is configured.
Expand Down Expand Up @@ -45,4 +46,6 @@ The `store` sub-directory contains the full software stack installation tree.
### SquashFS installation

The `store.squashfs` file is a compressed [SquashFS](https://tldp.org/HOWTO/SquashFS-HOWTO/whatis.html) image of the contents of the `store` path.
This can be mounted at runtime using [`squashfs-mount`](https://github.com/eth-cscs/squashfs-mount) or the [Slurm plugin](https://github.com/eth-cscs/slurm-uenv-mount/), or mounted by a system-administrator using [`mount`](https://man7.org/linux/man-pages/man2/mount.2.html), in order the to take advantage of the benefits of SquashFS over shared file systems.
This can be mounted at runtime using [`uenv`](https://github.com/eth-cscs/uenv) or the uenv Slurm plugin.

Images can also be mounted by a system-administrator using [`mount`](https://man7.org/linux/man-pages/man2/mount.2.html), which is used in production at CSCS to permanantly mount software stacks used by the weather service on their operational cluster when nodes boot.
Loading
Loading