Suggestion Description
Summary
Add support for Kubernetes Dynamic Resource Allocation (DRA) as an allocation path for AMD AINIC resources, alongside the existing classic device plugin (k8s-network-device-plugin). DRA's claims-based model would let workloads express topology and sharing constraints (NUMA/PCIe locality, multi-node fabric grouping, shared access) that the flat, count-based device plugin API structurally cannot express. More importantly, DRA would allow sharing the AINIC resource among containers.
Current state (verified against code)
The AMD Network Operator's device allocation path (k8s-network-device-plugin, an SR-IOV-based device plugin implementing k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1) today does the following, and no more:
- Flat resource counting. NICs/VFs are registered as allocatable resources (e.g.
amd.com/nic, amd.com/vnic) with a plain integer count per node. A pod requests "N of resource X"; there is no way to request "the NIC closest to GPU 2" or "a NIC on the same RDMA fabric segment as my peer pods."
- NUMA-node hint only, not PCIe-distance-aware.
pkg/devices/host.go reads /sys/bus/pci/<addr>/numa_node (pkg/utils/utils.go:GetDevNode) and attaches it as a single-node pluginapi.TopologyInfo (pkg/devices/api.go:NewAPIDeviceImpl). This is the stock kubelet device-plugin Topology field. It lets kubelet's built-in TopologyManager do coarse same-NUMA-node alignment across device plugins if that policy is enabled cluster-wide, but there is no PCIe bus/switch distance calculation and no pairing logic between a NIC and a specific GPU.
Allocate() is a pass-through. pkg/resources/server.go:Allocate just returns device specs/mounts/envs for whatever DeviceIDs kubelet already selected. There is no constraint-solving, scoring, or preference logic at allocation time.- No cross-node coordination anywhere in the operator. Per
docs/overview.md, the Network Operator's components (Controller Manager, NFD, device plugin, Node Labeller, Metrics Exporter, Multus/CNI, KMM-based driver management) are all per-node scoped. Multi-node RDMA connectivity for RCCL (per docs/mpi_rccl/mpi_rccl.md) is provided entirely by the external fabric (RoCE/InfiniBand) and by RCCL's own topology detection at the application layer. The operator has no concept of a "fabric domain" or multi-node interconnect group as a schedulable/claimable unit.
In short, today a pod can ask for "2 AINICs," but it cannot ask for "2 AINICs, each nearest on the PCIe bus to a specific GPU allocated to this pod" or "AINICs that belong to the same RDMA fabric partition as my sibling pods on other nodes." It also cannot share one physical NIC across multiple containers, except by pre-sizing the SR-IOV VF pool.
Why DRA
Classic device plugins expose a flat resource count with no way to express topology or sharing constraints. DRA's DeviceClass / ResourceClaim / ResourceClaimTemplate model lets a driver publish rich device attributes (PCIe path, NUMA node, fabric/rack grouping, capabilities) and lets claims select and share devices against those attributes via CEL expressions. This is also the direction upstream Kubernetes is moving as classic device plugins age out.
Use cases this would unblock
1. NUMA/PCIe-aware GPU-NIC pairing (single node)
Multi-GPU, multi-NIC training nodes need the NIC nearest a given GPU on the PCIe bus to hit full GPUDirect RDMA bandwidth, not just "same NUMA node" but ideally same PCIe switch/root complex. Today's device plugin can't express this. kubelet's TopologyManager alignment (when enabled) is NUMA-node granularity only, and there's no guarantee the GPU device plugin and this NIC device plugin even agree on a pairing.
2. Shared NIC access between containers
DRA's claim model supports sharing one physical device across multiple containers via a shared ResourceClaim. The gap here: today, sharing one physical AINIC across multiple pods/containers on a node is only possible by pre-provisioning a fixed number of SR-IOV VFs at NetworkConfig time, which hard-caps concurrency and wastes VFs when actual per-pod bandwidth needs are small. A shared-claim model would let multiple containers reference the same underlying device/queue set dynamically, sized to actual demand rather than a static VF count.
3. Multi-node fabric/domain-aware NIC allocation
GPU-to-GPU interconnect (xGMI) is intra-node for AMD GPUs, but the inter-node interconnect for RCCL collectives is exactly what this operator's AINIC fabric provides, and today that fabric has no representation as a schedulable/claimable unit. A DRA driver for AINIC could introduce a domain/group construct so a distributed job's pods, across nodes, all claim NICs belonging to the same declared RDMA fabric partition/rail, instead of relying on best-effort external fabric layout with no scheduling-time guarantee.
4. (Follow-on) Finer allocation constraints
Once device attributes are published via DRA (PCIe path, NUMA node, fabric group, queue/VF capacity, RDMA namespace), DeviceClass selectors could express constraints such as "min N queues," "same fabric rail as claim X," or "exclude devices already claimed by tenant Y," none of which are expressible through the current integer-resource model.
Proposed approach (for discussion)
- Add a DRA driver as an additional allocation path alongside the existing classic device plugin, not a replacement. Many clusters won't have DRA enabled, and the classic device plugin path should keep working.
- Publish per-device attributes beyond the current single NUMA-node topology hint: PCIe bus path (for locality scoring against GPU DRA drivers, if co-deployed), NUMA node, and (new) a fabric/domain identifier derived from the AINIC/switch topology, to support use case 3.
- Investigate a shared-claim mode for VF/queue-level sharing to support use case 2.
- Scope out minimum Kubernetes version / feature-gate requirements for DRA (structured parameters) that the operator would need to support, and how this interacts with the existing NFD/Controller Manager reconciliation model.
Open questions
- Does AINIC hardware/driver expose enough topology metadata (PCIe path, switch/rail grouping) today to populate these attributes accurately, or does that require driver-side work first?
- Should the multi-node fabric-domain construct be modeled operator-side (a new CRD, analogous to
NetworkConfig) or purely as DRA device attributes consumed by claims?
- What's the minimum supported Kubernetes/DRA API version this project is willing to take a dependency on?
References
Operating System
Common Supported Linux distributions
GPU
No response
ROCm Component
Network Operator (specifically k8s-network-device-plugin)
Suggestion Description
Summary
Add support for Kubernetes Dynamic Resource Allocation (DRA) as an allocation path for AMD AINIC resources, alongside the existing classic device plugin (
k8s-network-device-plugin). DRA's claims-based model would let workloads express topology and sharing constraints (NUMA/PCIe locality, multi-node fabric grouping, shared access) that the flat, count-based device plugin API structurally cannot express. More importantly, DRA would allow sharing the AINIC resource among containers.Current state (verified against code)
The AMD Network Operator's device allocation path (
k8s-network-device-plugin, an SR-IOV-based device plugin implementingk8s.io/kubelet/pkg/apis/deviceplugin/v1beta1) today does the following, and no more:amd.com/nic,amd.com/vnic) with a plain integer count per node. A pod requests "N of resource X"; there is no way to request "the NIC closest to GPU 2" or "a NIC on the same RDMA fabric segment as my peer pods."pkg/devices/host.goreads/sys/bus/pci/<addr>/numa_node(pkg/utils/utils.go:GetDevNode) and attaches it as a single-nodepluginapi.TopologyInfo(pkg/devices/api.go:NewAPIDeviceImpl). This is the stock kubelet device-pluginTopologyfield. It lets kubelet's built-inTopologyManagerdo coarse same-NUMA-node alignment across device plugins if that policy is enabled cluster-wide, but there is no PCIe bus/switch distance calculation and no pairing logic between a NIC and a specific GPU.Allocate()is a pass-through.pkg/resources/server.go:Allocatejust returns device specs/mounts/envs for whateverDeviceIDskubelet already selected. There is no constraint-solving, scoring, or preference logic at allocation time.docs/overview.md, the Network Operator's components (Controller Manager, NFD, device plugin, Node Labeller, Metrics Exporter, Multus/CNI, KMM-based driver management) are all per-node scoped. Multi-node RDMA connectivity for RCCL (perdocs/mpi_rccl/mpi_rccl.md) is provided entirely by the external fabric (RoCE/InfiniBand) and by RCCL's own topology detection at the application layer. The operator has no concept of a "fabric domain" or multi-node interconnect group as a schedulable/claimable unit.In short, today a pod can ask for "2 AINICs," but it cannot ask for "2 AINICs, each nearest on the PCIe bus to a specific GPU allocated to this pod" or "AINICs that belong to the same RDMA fabric partition as my sibling pods on other nodes." It also cannot share one physical NIC across multiple containers, except by pre-sizing the SR-IOV VF pool.
Why DRA
Classic device plugins expose a flat resource count with no way to express topology or sharing constraints. DRA's
DeviceClass/ResourceClaim/ResourceClaimTemplatemodel lets a driver publish rich device attributes (PCIe path, NUMA node, fabric/rack grouping, capabilities) and lets claims select and share devices against those attributes via CEL expressions. This is also the direction upstream Kubernetes is moving as classic device plugins age out.Use cases this would unblock
1. NUMA/PCIe-aware GPU-NIC pairing (single node)
Multi-GPU, multi-NIC training nodes need the NIC nearest a given GPU on the PCIe bus to hit full GPUDirect RDMA bandwidth, not just "same NUMA node" but ideally same PCIe switch/root complex. Today's device plugin can't express this. kubelet's
TopologyManageralignment (when enabled) is NUMA-node granularity only, and there's no guarantee the GPU device plugin and this NIC device plugin even agree on a pairing.2. Shared NIC access between containers
DRA's claim model supports sharing one physical device across multiple containers via a shared
ResourceClaim. The gap here: today, sharing one physical AINIC across multiple pods/containers on a node is only possible by pre-provisioning a fixed number of SR-IOV VFs atNetworkConfigtime, which hard-caps concurrency and wastes VFs when actual per-pod bandwidth needs are small. A shared-claim model would let multiple containers reference the same underlying device/queue set dynamically, sized to actual demand rather than a static VF count.3. Multi-node fabric/domain-aware NIC allocation
GPU-to-GPU interconnect (xGMI) is intra-node for AMD GPUs, but the inter-node interconnect for RCCL collectives is exactly what this operator's AINIC fabric provides, and today that fabric has no representation as a schedulable/claimable unit. A DRA driver for AINIC could introduce a domain/group construct so a distributed job's pods, across nodes, all claim NICs belonging to the same declared RDMA fabric partition/rail, instead of relying on best-effort external fabric layout with no scheduling-time guarantee.
4. (Follow-on) Finer allocation constraints
Once device attributes are published via DRA (PCIe path, NUMA node, fabric group, queue/VF capacity, RDMA namespace),
DeviceClassselectors could express constraints such as "min N queues," "same fabric rail as claim X," or "exclude devices already claimed by tenant Y," none of which are expressible through the current integer-resource model.Proposed approach (for discussion)
Open questions
NetworkConfig) or purely as DRA device attributes consumed by claims?References
Operating System
Common Supported Linux distributions
GPU
No response
ROCm Component
Network Operator (specifically k8s-network-device-plugin)