Move kmod-6.18 nvidia drivers to use overlayfs at boot

[vigh-m]

Description of changes:
Enables the NVIDIA lts (r580) and pb (r595) driver branches to be co-installed on a single Bottlerocket image, with exactly one branch activated at boot. Single-driver images continue to work unchanged.

Both branches previously installed their libraries, binaries, module objects and configs directly onto the canonical system paths (/usr/lib64, /usr/bin, /usr/share/nvidia, /etc/nvidia, ...). This PR moves each branch's files under a per-branch storage root and overlays the selected branch onto the canonical paths at boot.

• Each branch installs into usr/share/.nvidia/<branch>/ (lts or pb).
• Per-branch, oneshot services mount overlays that make the storage root's contents appear at the canonical paths:
  • nvidia-<branch>-overlay-driver.service: lib64, bin, share/nvidia, share/vulkan, share/glvnd, share/egl.
  • nvidia-<branch>-overlay-config.service: /etc/nvidia, /etc/nvidia-imex.
• The /usr/lib64 overlay would otherwise shadow the writable /usr/lib/modules mount, so the base module tree is bound to a /run mount point and re-layered as a lowerdir with a writable upper under /var/lib/kernel-modules/.nvidia.
• driverdog link/load/copy unit filenames are namespaced per branch (nvidia-lts-* / nvidia-pb-*) so both branches' units coexist without collision. driverdog config tomls are also branch-named and installed normally into the flat /etc/drivers.

New driver file layout

Each branch now lives under its own root, e.g.:

[root@admin]# tree /.bottlerocket/rootfs/usr/share/.nvidia/ -d
/.bottlerocket/rootfs/usr/share/.nvidia/
|-- lts
|   |-- bin
|   |-- etc
|   |   |-- nvidia
|   |   `-- nvidia-imex
|   |-- lib64
|   |   `-- tesla
|   `-- share
|       |-- egl
|       |   `-- egl_external_platform.d
|       |-- glvnd
|       |   `-- egl_vendor.d
|       |-- nvidia
|       |   |-- grid
|       |   |   `-- drivers
|       |   |-- open-gpu
|       |   |   `-- drivers
|       |   `-- tesla
|       |       |-- module-objects.d
|       |       `-- nvswitch
|       `-- vulkan
|           |-- icd.d
|           `-- implicit_layer.d
`-- pb
    |-- bin
    |-- etc
    |   |-- nvidia
    |   `-- nvidia-imex
    |-- lib64
    |   `-- tesla
    `-- share
        |-- egl
        |   `-- egl_external_platform.d
        |-- glvnd
        |   `-- egl_vendor.d
        |-- nvidia
        |   |-- grid
        |   |   `-- drivers
        |   |-- open-gpu
        |   |   `-- drivers
        |   `-- tesla
        |       |-- module-objects.d
        |       `-- nvswitch
        `-- vulkan
            |-- icd.d
            `-- implicit_layer.d

Backwards compatibility

• A single-driver image installs exactly one branch; its overlay services project that branch onto the canonical paths, giving the same runtime layout as before.

Testing

• Single-driver lts (r580) image: boots, driver loads, nvidia-smi works
• Single-driver pb (r595) image: boots, driver loads, nvidia-smi works
• Overlay mounts present after boot (lib64/bin/share-nvidia/etc-nvidia)
• Nvidia Smoke test workload on g4dn, g5, g6, g7(595 only)
• MIG and EFA testing (580 and 595)

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[qianxjcraig]

Could we refresh the host linker cache after mounting the selected driver's overlays?

On dual-branch boots, the existing ldconfig.service runs before branch selection, so its scan cannot see these NVIDIA libraries. The legacy NVIDIA runtime uses that host cache to discover libraries to mount into containers.

Appending the refresh here would keep the existing early ldconfig.service in place and update the cache after the overlays are mounted.

Suggested change

	ExecStart=/usr/bin/mount -t overlay overlay --options ro,nosuid,nodev,noexec,noatime,lowerdir=__NVIDIA_ROOT__/share/egl:/usr/share/egl /usr/share/egl
	ExecStart=/usr/bin/mount -t overlay overlay --options ro,nosuid,nodev,noexec,noatime,lowerdir=__NVIDIA_ROOT__/share/egl:/usr/share/egl /usr/share/egl
	ExecStart=/sbin/ldconfig -X

The PB overlay service needs the same change. Could we also validate this with a legacy-mode GPU container test?

[vigh-m]

Yeah. This isn't really needed. Since we overlay the .so files into /usr/lib64 glibc will fallback to that. Running ldconfig only populates the ldconf.so.cache which has almost no impact on a customers workflow.

When testing by calling nvidia-smi, the binary was discovered as expected so this is safe to omit.