Skip to content

feat(operator): Add node capacity labeler for GPU/Neuron - #412

Draft
movence wants to merge 1 commit into
mainfrom
feat/node-capacity-labeler
Draft

movence wants to merge 1 commit into
mainfrom
feat/node-capacity-labeler

Conversation

@movence

@movence movence commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Description

Add a NodeLabeler controller that watches Nodes and mirrors advertised accelerator capacity into fixed-key labels a DaemonSet affinity can select on, since the scheduler cannot match on status.capacity directly.

  • Presence-of-key semantics: stamps cloudwatch.aws.amazon.com/gpu.present=true when status.capacity has the nvidia.com/gpu key, and cloudwatch.aws.amazon.com/neuron.present=true for any of aws.amazon.com/neuron, /neuroncore, /neurondevice. The label keys on the capacity key existing, not its quantity, so a GPU node whose device plugin is unhealthy (kubelet keeps the key at quantity 0) stays labeled.
  • Labels-only merge patch: reconcile applies a JSON merge patch scoped to metadata.labels only (never the full object), so it never races the kubelet's frequent status writes; it is a no-op when labels are already correct, and a predicate filters out irrelevant node status churn.
  • Feature flag: gated by --enable-node-capacity-labeler, default true.
  • RBAC: adds nodes get;list;watch;patch, generated from a kubebuilder marker (zero make manifests drift).

Related

A companion helm-charts change to point the dcgm-exporter DaemonSet affinity at cloudwatch.aws.amazon.com/gpu.present (replacing the hardcoded instance-type list) is a follow-up, not part of this PR.

Testing

  • Unit: 8 table-style tests covering GPU/Neuron labeling, presence-of-key (quantity 0) semantics, stale-label removal, no-op when already correct, other-label preservation, and the event predicate — go test ./controllers/... -run NodeLabeler -v, all pass. go build ./... and go vet ./... clean.
  • Manual (a test EKS cluster with one g4dn GPU node):
    • The label appeared only on the GPU node; that node's existing labels were left intact.
    • A dcgm-exporter DaemonSet using label-only affinity scheduled from 0 -> 1 pod onto the GPU node.
    • A label manually added to a non-GPU node was removed by the operator within ~1s.
    • No patch storm over several minutes of steady state.
    • DCGM GPU metrics reached CloudWatch.
  • Defect fixed during bring-up: the manager cache set ByObject[Node].Namespaces, which controller-runtime rejects for cluster-scoped types at startup; removed it, since cluster-scoped kinds are always watched cluster-wide.

Worth noting

  • There is a short boot-time lag before a fresh GPU node is labeled, since the label follows the device plugin registering capacity.
  • The neuron label is stamped, but the chart's neuronMonitor affinity is not switched to it here.
  • There is no chart value yet to disable the labeler (the operator flag defaults on).

Add a NodeLabeler controller that watches Nodes and mirrors accelerator
capacity into fixed-key labels a DaemonSet affinity can select on, since
the scheduler cannot match on status.capacity directly.

It stamps cloudwatch.aws.amazon.com/gpu.present=true when status.capacity
has the key nvidia.com/gpu, and cloudwatch.aws.amazon.com/neuron.present
=true when it has any of aws.amazon.com/neuron, /neuroncore, or
/neurondevice. The label keys on presence of the capacity key, not its
quantity, so a GPU node whose device plugin is unhealthy (kubelet resets
the quantity to 0 but keeps the key) stays labeled.

Reconcile applies a JSON merge patch scoped to metadata.labels only,
avoiding races with the kubelet's frequent status updates, and is a
no-op when the labels are already correct. A predicate filters out
irrelevant node status churn so only capacity or managed-label changes
trigger work.

Gated by the --enable-node-capacity-labeler flag (default true). Adds
RBAC for nodes get/list/watch/patch. This lets the dcgm-exporter/neuron
DaemonSet affinity select a stable label instead of a hardcoded
instance-type list.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant