Skip to content

packages: add nvidia-dra-driver-gpu - #1035

Merged
maherthomsi merged 2 commits into
bottlerocket-os:developfrom
maherthomsi:dra-changes
Sep 18, 2026
Merged

maherthomsi merged 2 commits into
bottlerocket-os:developfrom
maherthomsi:dra-changes

Conversation

@maherthomsi

@maherthomsi maherthomsi commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Description of changes:
Adds the nvidia-dra-driver-gpu package, which builds the NVIDIA GPU DRA driver (gpu-kubelet-plugin) and installs it as a host systemd service. The package is added to the bottlerocket-core-kit workspace and included in the kit manifest so it is available to consuming variants.

Bump settings sdk for this change to include bottlerocket-os/bottlerocket-settings-sdk#148

The DRA driver (nvidia-dra-driver-gpu.service) and the kubernetes device plugin (nvidia-k8s-device-plugin.service) both allocate the same GPUs, so only one runs at a time. This is enforced without a systemd Conflicts=: the DRA unit gates on ConditionFileNotEmpty=/etc/nvidia-dra-driver-gpu/enabled, and the marker is rendered run (non-empty) only when DRA is enabled and the device plugin is disabled. DRA is therefore opt-in and off by default.

Both settings present

kubelet-dra-drivers.nvidia.enabled kubelet-device-plugins.nvidia.enabled DRA service device-plugin service GPU publisher
enabled disabled running off DRA (ResourceSlice)
enabled enabled off (defers) running device plugin (nvidia.com/gpu)
disabled enabled off running device plugin
disabled disabled off off none

Settings extension not present (failure to adopt the DRA settings API)

Condition DRA marker DRA service device-plugin service GPU publisher
kubelet-dra-drivers extension absent not rendered off running (its own default) device plugin
kubelet-dra-drivers extension absent not rendered off not running (disabled via API) DRA

What "unset" means per setting

Setting If unset Effect
kubelet-dra-drivers.nvidia.enabled treated as disabled (default "false"; shipped default is false) DRA off
kubelet-device-plugins.nvidia.enabled treated as enabled (default "true"; classic behavior) device plugin runs

Out-of-the-box default (both unset): DRA off, device plugin running.

Guarantees: exactly one GPU publisher at any time; the device plugin wins when both are enabled (deterministic, no boot-time race); with no settings API the DRA driver stays off

Testing done:

  • Built the kits for x86_64 and aarch64
  • Launched a custom ami with the driver added to the variant and ensured that the DRA driver was loaded.
    • Tested on aarch64
    • Tested on x86_64

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu.spec Outdated
Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu.spec Outdated
Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu.spec Outdated
Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu.spec Outdated
Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu.spec Outdated
Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu.service Outdated
Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu-exec-start-conf Outdated
@maherthomsi

Copy link
Copy Markdown
Contributor Author

Added settings-sdk changes to this PR

@maherthomsi
maherthomsi force-pushed the dra-changes branch 4 times, most recently from 63c65b5 to 5dc5941 Compare September 11, 2026 22:06
Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu-exec-start-conf Outdated
@maherthomsi

maherthomsi commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Addresses review feedback on the nvidia-dra-driver-gpu package:

  • spec: add Requires: %{_cross_os}nvidia-container-toolkit - the rendered unit invokes /usr/bin/nvidia-cdi-hook, which that package ships.
  • service (base unit): replace the ExecStart=/usr/bin/false + Type=simple + Restart=always placeholder with an inert no-op (Type=oneshot, RemainAfterExit=true, ExecStart=/usr/bin/true). It no longer retries forever / logs to console before the drop-in renders, and the ExecStartPre checks are no longer duplicated between the base unit and the drop-in.
  • exec-start-conf (drop-in): reference the optional settings.kubernetes.hostname-override only inside {{#if_not_null}}, so an unset value no longer fails the strict render. Move the daemon's Type=simple/Restart=always/RestartSec/RemainAfterExit=false into the enabled branch (previously inherited from the base) and drop the redundant {{else}} branch since the inert base now covers the disabled/unset case.

Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu-exec-start-conf Outdated
Comment thread packages/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu.service
@maherthomsi
maherthomsi force-pushed the dra-changes branch 2 times, most recently from 93457dd to 1919b69 Compare September 12, 2026 01:02
piyush-jena
piyush-jena previously approved these changes Sep 14, 2026
@maherthomsi
maherthomsi force-pushed the dra-changes branch 4 times, most recently from 41ee794 to 7de8784 Compare September 16, 2026 23:40
Add the nvidia-dra-driver-gpu package, which builds the NVIDIA GPU DRA
driver (gpu-kubelet-plugin) and installs it as a host systemd service.
Include it in the bottlerocket-core-kit workspace and kit manifest.

Signed-off-by: Maher Homsi <maherhom@amazon.com>
Signed-off-by: Maher Homsi <maherhom@amazon.com>
@piyush-jena
piyush-jena dismissed their stale review September 17, 2026 17:55

new changes pushed. I will review them.

@piyush-jena

piyush-jena commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Looks good to me!

I had this design in mind which I think should work:
nvidia-k8s-device-plugin.service creates a marker file (nvidia-k8s-device-plugin.ran) after running. nvidia-dra-driver-plugin.service runs after/requires nvidia-k8s-device-plugin.service, if it sees the marker file then doesn't run, if it doesn't see the marker file, runs based on the settings.

The design I described comes with a very small boot time penalty but imo is cleaner. Your design works too.

@maherthomsi
maherthomsi merged commit f1f405f into bottlerocket-os:develop Sep 18, 2026
5 checks passed
@maherthomsi
maherthomsi deleted the dra-changes branch September 18, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants