Conversation
Add a NodeLabeler controller that watches Nodes and mirrors accelerator capacity into fixed-key labels a DaemonSet affinity can select on, since the scheduler cannot match on status.capacity directly. It stamps cloudwatch.aws.amazon.com/gpu.present=true when status.capacity has the key nvidia.com/gpu, and cloudwatch.aws.amazon.com/neuron.present =true when it has any of aws.amazon.com/neuron, /neuroncore, or /neurondevice. The label keys on presence of the capacity key, not its quantity, so a GPU node whose device plugin is unhealthy (kubelet resets the quantity to 0 but keeps the key) stays labeled. Reconcile applies a JSON merge patch scoped to metadata.labels only, avoiding races with the kubelet's frequent status updates, and is a no-op when the labels are already correct. A predicate filters out irrelevant node status churn so only capacity or managed-label changes trigger work. Gated by the --enable-node-capacity-labeler flag (default true). Adds RBAC for nodes get/list/watch/patch. This lets the dcgm-exporter/neuron DaemonSet affinity select a stable label instead of a hardcoded instance-type list.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add a
NodeLabelercontroller that watches Nodes and mirrors advertised accelerator capacity into fixed-key labels a DaemonSet affinity can select on, since the scheduler cannot match onstatus.capacitydirectly.cloudwatch.aws.amazon.com/gpu.present=truewhenstatus.capacityhas thenvidia.com/gpukey, andcloudwatch.aws.amazon.com/neuron.present=truefor any ofaws.amazon.com/neuron,/neuroncore,/neurondevice. The label keys on the capacity key existing, not its quantity, so a GPU node whose device plugin is unhealthy (kubelet keeps the key at quantity 0) stays labeled.metadata.labelsonly (never the full object), so it never races the kubelet's frequent status writes; it is a no-op when labels are already correct, and a predicate filters out irrelevant node status churn.--enable-node-capacity-labeler, defaulttrue.nodesget;list;watch;patch, generated from a kubebuilder marker (zeromake manifestsdrift).Related
A companion helm-charts change to point the dcgm-exporter DaemonSet affinity at
cloudwatch.aws.amazon.com/gpu.present(replacing the hardcoded instance-type list) is a follow-up, not part of this PR.Testing
go test ./controllers/... -run NodeLabeler -v, all pass.go build ./...andgo vet ./...clean.ByObject[Node].Namespaces, which controller-runtime rejects for cluster-scoped types at startup; removed it, since cluster-scoped kinds are always watched cluster-wide.Worth noting