Skip to content

Remove backwards symlinks for 1.37+ - #4942

Merged
maherthomsi merged 2 commits into
bottlerocket-os:developfrom
maherthomsi:remove-backwards-symlinks
Sep 19, 2026
Merged

maherthomsi merged 2 commits into
bottlerocket-os:developfrom
maherthomsi:remove-backwards-symlinks

Conversation

@maherthomsi

@maherthomsi maherthomsi commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Description of changes:

Starting in EKS k8s 1.37, nvidia variants no longer create the legacy /usr/lib/nvidia/tesla/ backwards-compatibility symlinks inside containers. Libraries live at the FHS-standard /usr/lib/ only. Variants for k8s 1.36 and below keep the legacy path so existing customer workloads are unaffected.

The companion core-kit change makes the clean, no-symlink templates the default and ships -compat templates alongside them. This PR wires up the variant settings-defaults to select behavior per variant:

  • shared-defaults: add nvidia tesla backwards-compat symlink defaults — adds two settings fragments:
    • nvidia-k8s-device-plugin-compat-symlinks.toml renders the -compat device-plugin exec-start template, which keeps --cdi-enabled-hooks create-lib-symlinks (the path used by ordinary GPU pods).
    • nvidia-cdi-specs-compat-symlinks.toml renders a generate-cdi-specs.service drop-in that adds --additional-symlinks /usr/lib/nvidia/tesla (the static CDI spec path used by ECS and the k8s NVIDIA_VISIBLE_DEVICES=all path). Its restart-command is systemctl daemon-reload (not a restart) because the unit sets RefuseManualStart=true; the unit is ordered After=settings-applier.service so the drop-in is on disk before the oneshot runs.
  • settings-defaults: keep nvidia tesla symlinks on pre-1.37 variants — includes these fragments in all existing nvidia variants so their behavior is unchanged:
    • k8s 1.31–1.36 nvidia (and -fips): both the device-plugin and generate-cdi-specs fragments.
    • ECS 2/3/4 and mantle-1 nvidia (and -fips): the generate-cdi-specs fragment only (no device plugin).

The new aws-k8s-1.37-nvidia / aws-k8s-1.37-nvidia-fips variants intentionally omit these fragments, so they inherit the clean default templates and produce no /usr/lib/nvidia/tesla/ symlinks — the desired end state.

Testing done:
Built and deployed an aws-k8s-1.36-nvidia AMI (A10G / R580 580.178.04, g5.xlarge, EKS 1.36 cluster) and
verified from ground-truth on-host sources:

  • Host: nvidia-smi works; NVIDIA libs present in /usr/lib/
  • Host: CDI spec generated at /etc/cdi/nvidia.json
  • Device plugin registers with kubelet (node advertises nvidia.com/gpu), CDI mode
  • With compat fragments (1.36): rendered device-plugin exec-start.conf includes --cdi-enabled-hooks
    create-lib-symlinks
  • With compat fragments (1.36): /etc/cdi/nvidia.json contains nvidia/tesla references (legacy path
    preserved)
  • Clean default templates confirmed present/used when compat fragments are absent (demonstrates 1.37+
    no-symlink behavior)
  • Live GPU pod (nvidia-smi in-pod; inspect in-container /usr/lib/nvidia/tesla)
  • Built aws-k8s-1.37-nvidia AMI — confirm no container-side tesla symlinks end-to-end

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@maherthomsi
maherthomsi force-pushed the remove-backwards-symlinks branch from 263f0af to 48b6d0c Compare September 17, 2026 20:23
@maherthomsi
maherthomsi marked this pull request as ready for review September 17, 2026 22:10
@maherthomsi maherthomsi changed the title Remove backwards symlinks Remove backwards symlinks for 1.37+ Sep 18, 2026

@ginglis13 ginglis13 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm just some edits to verbose comments

Comment thread sources/shared-defaults/nvidia-cdi-specs-compat-symlinks.toml Outdated
Comment thread sources/shared-defaults/nvidia-k8s-device-plugin-compat-symlinks.toml Outdated
@maherthomsi
maherthomsi force-pushed the remove-backwards-symlinks branch from 48b6d0c to 565b725 Compare September 18, 2026 22:18
maherthomsi added a commit to maherthomsi/bottlerocket that referenced this pull request Sep 18, 2026
Add two settings fragments used to keep the legacy
/usr/lib/nvidia/tesla symlinks on nvidia variants that predate the
EKS k8s 1.37 removal:

- nvidia-k8s-device-plugin-compat-symlinks.toml overrides the
  device-plugin exec-start template with the -compat variant, which
  keeps create-lib-symlinks.
- nvidia-cdi-specs-compat-symlinks.toml renders a generate-cdi-specs
  drop-in that adds --additional-symlinks, with a daemon-reload
  restart-command so the drop-in is loaded before the oneshot runs
  (the unit sets RefuseManualStart).

Signed-off-by: Maher Homsi <maherhom@amazon.com>
Include the nvidia backwards-compat symlink defaults in all existing
nvidia variants so their behavior is unchanged; only future EKS k8s
1.37+ variants omit them. k8s 1.31-1.36 get both the device-plugin
and generate-cdi-specs fragments; ECS and mantle variants get the
generate-cdi-specs fragment only (no device plugin).

Signed-off-by: Maher Homsi <maherhom@amazon.com>
@maherthomsi
maherthomsi force-pushed the remove-backwards-symlinks branch from 565b725 to de643b6 Compare September 19, 2026 01:54
@maherthomsi
maherthomsi merged commit e797726 into bottlerocket-os:develop Sep 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants