Skip to content

test(otel): add GPU, Neuron, and EFA DRA-path integration tests - #753

Open
samehkhalil wants to merge 1 commit into
aws:mainfrom
samehkhalil:test/multi-efa-dra-per-device-correlation
Open

samehkhalil wants to merge 1 commit into
aws:mainfrom
samehkhalil:test/multi-efa-dra-per-device-correlation

Conversation

@samehkhalil

Copy link
Copy Markdown

Description of the issue

EFA metrics on EKS can be exposed to pods two ways: the EFA device plugin
(vpc.amazonaws.com/efa) and Dynamic Resource Allocation (DRA), where EFA
devices are allocated via a DRA driver (dranet, driver dra.net) and
ResourceClaims. The agent's OTel Container Insights pipeline supports both, and
the awsdevicepodcorrelation processor has a dedicated DRA code path that watches
ResourceClaims/ResourceSlices via the K8s API and bridges the DRA device
identity (a PCI name, e.g. pci-0000-00-1e-0) to the EFA metric label (e.g.
rdmap0s30) via the dra.net/rdmaDevice ResourceSlice attribute.

Today there is no integration coverage for the DRA path — only the device-plugin
path is exercised. This adds an end-to-end test so the DRA path is validated on a
real cluster: EFA metrics are emitted per device and correctly correlated to the
pods that claim them.

Description of changes

Add an integration test that provisions a cluster exposing EFA via DRA and
validates per-device EFA metric correlation end to end:

  • test/otel/multi_efa_dra/ — queries the EFA metrics in CloudWatch and
    asserts the DRA path produces the expected per-device series and pod
    attribution: each device is a distinct series with its EFA attributes
    (aws.efa.device, ENI, port), the device a pod claims is correlated to that
    pod (name/namespace/container), and devices not claimed by any pod carry no pod
    attributes.
  • terraform/eks/daemon/otel-multi-efa-dra/ — provisions the cluster,
    installs dranet (eks/aws-dranet) as the DRA driver, deploys the
    observability chart and agent, and runs an efaburn workload that requests one
    EFA through a ResourceClaimTemplate.
  • Generator entry wiring the test to its terraform module.
  • helm_chart_repo_url variable so the chart can be cloned from a fork when
    validating chart changes not yet merged upstream (defaults to upstream).

Pinned to k8s 1.34 (the rest of the suite is on 1.35): the processor's DRA
informers watch resource.k8s.io/v1beta1, which 1.34 still serves alongside the
GA v1. The generator entry is commented with this rationale; it moves to 1.35
once the processor's DRA client is bumped to v1.

Dependencies (the EKS lane is green only after these land): the chart on
main must render the DRA correlation config (dra_device_types on the
dra.net driver + dra.net/rdmaDevice keying) and grant the agent ServiceAccount
get/list/watch on resource.k8s.io resourceclaims/resourceslices, and the
released agent image must include the DRA processor path. Committed defaults
already point at that merged end-state (public agent image, chart main, upstream
chart URL), so no follow-up edit is needed once those merge.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Ran the module end to end on a live cluster (EKS 1.34, c6in.32xlarge, EFA via
dranet). All tests passed.

Verified against ground truth:

  • The node exposed 2 EFA devices, rdmap0s30 and rdmap0s31.
  • efaburn's ResourceClaimTemplate allocated pci-0000-00-1e-0, which maps via
    dra.net/rdmaDevice to rdmap0s30 → correlated to the efaburn pod
    (namespace default, container efaburn).
  • rdmap0s31 was unclaimed → no pod attributes.

Each EFA metric (efa_rx_bytes, efa_tx_bytes, efa_rx_dropped,
efa_rdma_read_bytes) reported per device with correct DRA-based pod correlation.
Cluster torn down after the run.

@samehkhalil
samehkhalil requested a review from a team as a code owner September 2, 2026 11:53
@samehkhalil
samehkhalil force-pushed the test/multi-efa-dra-per-device-correlation branch from 077c30e to f803c04 Compare September 8, 2026 16:28
@samehkhalil samehkhalil changed the title test(otel/multi_efa_dra): add integration test for the EFA DRA path test(otel): add GPU, Neuron, and EFA DRA-path integration tests Sep 8, 2026
@samehkhalil
samehkhalil force-pushed the test/multi-efa-dra-per-device-correlation branch from f803c04 to d43ae5d Compare September 8, 2026 16:56
Add integration coverage for the awsdevicepodcorrelation processor's DRA
(Dynamic Resource Allocation) path, mirroring the device-plugin GPU/Neuron/EFA
correlation tests. Each package exposes its devices via a DRA driver (through a
ResourceClaimTemplate) instead of the device-plugin resource, and asserts
per-device pod correlation.

- test/otel/multi_efa_dra: EFA via dranet (driver dra.net); efaburn claims one
  of two devices, the other stays unclaimed. Guards the per-device correlation
  collapse (ResourceSlice keying via dra.net/rdmaDevice plus the groupbyattrs
  split before the resource-level promote).
- test/otel/neuron_dra: Neuron via the AWS Neuron DRA driver (DeviceClass
  neuron.aws.com). Single Trainium device (trn1.2xlarge); the claimed device's
  two cores attribute to the burn pod and to no other pod. The Neuron DRA driver
  supports Trainium only, so this targets trn1.2xlarge.
- test/otel/gpu_dra: GPU via the NVIDIA DRA driver (DeviceClass gpu.nvidia.com).
  g4dn.12xlarge (4 GPUs); one claimed GPU correlates to the burn pod, the other
  three stay uncorrelated. Asserts device count, consecutive indices, and all
  DCGM metrics per device.

New terraform modules under terraform/eks/daemon (otel-multi-efa-dra,
otel-neuron-dra, otel-gpu-dra) install the DRA driver in place of the device
plugin and apply a ResourceClaimTemplate burn workload. The processor uses the
GA resource.k8s.io/v1 DRA API (available since Kubernetes 1.34), so the clusters
run k8s 1.35 like the rest of the suite. Wired into the test case generator.
Requires a chart carrying the DRA correlation config and resource.k8s.io RBAC.
@samehkhalil
samehkhalil force-pushed the test/multi-efa-dra-per-device-correlation branch from d43ae5d to 3073b10 Compare September 22, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant