fix: carry conda activate.d env hooks into conda/micromamba:v2 images

[pinin4fjords]

Summary

Switching the default Conda build template to conda/micromamba:v2 exposed a compatibility issue with Conda packages that rely on activate.d scripts.

The v2 template builds the Conda environment in one stage and copies it into a clean final image. Unlike the previous template, the final image does not run micromamba activate, so the environment variables configured by activate.d scripts are not loaded.

This affects packages such as cmdstan, which uses an activation script to set CMDSTAN, as well as compiler packages that set variables such as CC and CXX.

Fix

The v2 Docker and Singularity templates now:

1. Combine the installed activate.d scripts into a single script during the build.
2. Configure BASH_ENV in the final image so Bash loads those settings automatically.
3. Temporarily disable nounset while loading the scripts, since some activation scripts reference variables that may not yet be defined.

This restores the activation-script behavior for both Docker and Singularity images generated from the v2 templates.

Validation

• Updated the affected template tests, and added a test that runs the generated combiner script under real bash -ue to confirm exported variables survive and nounset is restored correctly, rather than only asserting on the rendered template text.
• Verified that cmdstan and compiler environment variables are available in generated images.
• Confirmed that a real CmdStan model can compile and run.
• Ran the existing template test suite successfully.

Fixes #1119

[pditommaso]

Thanks for the really thorough investigation here @pinin4fjords — the diagnosis is spot on, and cmdstan is a great concrete example. A few thoughts before we decide how to proceed. 🙂

On the framing — it's more a long-standing quirk than a v2 regression.
Worth noting that the v1 template never actually ran micromamba activate itself either. Its Docker images only picked up the activate.d hooks implicitly, by inheriting mambaorg/micromamba's own activation entrypoint — and v1 Singularity never activated at all, since singularity exec doesn't run a Docker ENTRYPOINT. So what v2 "dropped" was really an accidental Docker-only side effect rather than behavior we ever guaranteed. Not a blocker for the fix, just useful context for how we scope it.

I'd lean towards not changing the v2 default for this.
Since v2 is now the silent default for every Conda build, I'm a little cautious about baking activation semantics into it. Would either of these work for you instead?

• Sticking with the legacy v1 template for the affected builds, which keeps the inherited-entrypoint behavior on Docker, or
• Introducing this as a distinct v3 template, so it's opt-in and we don't shift the default behavior for everyone.

The combiner + BASH_ENV bit feels a touch hacky to live in the default template.
The generated script plus the nounset save/restore and the manual CONDA_PREFIX seeding are a fair few moving parts to embed in the template, and the sort of thing that could break quietly later. One alternative that might be simpler: avoid the multi-stage build and build straight on top of the micromamba image — then its normal activation entrypoint handles Docker with nothing template-side to maintain.

Fair warning that this last option only helps the Docker path — singularity exec still won't run the entrypoint, so Singularity would stay unactivated (same as today). If we're OK treating Singularity activation as out of scope, I think that's a cleaner route; happy to talk it through either way.

Really appreciate the work — just want to make sure we land it in the right place. 🙏

[pinin4fjords]

Thanks @pditommaso. I don't think this should be relegated to a legacy build version, it's a current and valid feature of conda environments as far as I know, so I think Wave and Seqera Containers should support it properly with a current version.

I chose this slightly hacky version specifically for the functionality across docker and singularity, having considered something along the lines of the tidier version you allude to. I don't think a solution that works only for Docker is the right call, but it's your call.

V3 works for me as long as we can get that deployed to Seqera containers soonish!