Skip to content

fix(metrics): keep k8s workload names when DNS resolves a destination late - #339

Merged
mayankpande88 merged 1 commit into
mainfrom
fix/fqdn-overwrites-resolved-workload
Sep 18, 2026
Merged

mayankpande88 merged 1 commit into
mainfrom
fix/fqdn-overwrites-resolved-workload

Conversation

@mangglesh

Copy link
Copy Markdown
Contributor

What this PR does

DestinationKey.WithResolvedDomain now leaves a side's name alone if that side already resolved to a k8s workload. Only unresolved or external sides get the FQDN.

This is the path that runs when the DNS cache fills after a connection was keyed. It replaced both workload names with the FQDN without checking. A connection whose ClusterIP hadn't resolved yet but whose pod had came out as:

actual_destination_workload_name = "rabbitmq.rabbit.svc.cluster.local"
actual_destination_workload_kind = "StatefulSet"

The service map then shows that as a second, phantom StatefulSet next to the real rabbitmq. #298 fixed the same problem in NewDestinationKey. This PR applies the same isKubernetesResolved check to the second path it missed.

Linked issue / context

Part of nudgebee/nudgebee-enterprise#35231 (phantom KG workloads). That issue has two causes:

How was this tested?

  • New TestWithResolvedDomain_ResolvedWorkloadKeepsItsName fails on main (gets rabbitmq.rabbit.svc.cluster.local, expects rabbitmq) and passes with the fix. TestWithResolvedDomain_ExternalTakesFQDN checks that external destinations still get the FQDN.
  • In golang:1.25 (Linux): go vet and go test pass for ./common/ and ./containers/, including a run with -race.
  • To check on a cluster: this query should return nothing:
    count({actual_destination_workload_name=~".*[.]svc[.]cluster[.]local", actual_destination_workload_kind!="external"}).
    It already returns 0 on dev-gke with v0.1.7. This path only triggers when the pod IP counts as "external" (outside RFC1918 and 100.64/10), which is common for Service ClusterIPs on GKE. So this closes a gap that can still happen, not a flood currently on dev.
Notes
  • Both callers (enrichDestinationKey, migrateConnectionKeyIfNeeded) check only the destination name before calling in. The actual side got overwritten no matter what kind it was.
  • One full-package -race run of ./common/ flagged TestEphemeralWorkloadAggregation. It passed 10/10 in isolation on both this branch and main, and a full-package -race rerun was clean. That test doesn't touch DestinationKey. It looks like a flaky test sharing global state, and CI doesn't run with -race.
  • CHANGELOG not updated: [Unreleased] isn't kept per fix in this repo.

eBPF changes?

No.

Checklist

  • gofmt / go vet pass
  • go test passes (./common/, ./containers/ on Linux)
  • CHANGELOG.md updated (not used per fix)
  • No real credentials, internal hostnames, or production captures in test fixtures

🤖 Generated with Claude Code

… late

WithResolvedDomain replaced both destination workload names with the
FQDN, even when a side had already resolved to a k8s workload. That
produced labels like name="rabbitmq.rabbit.svc.cluster.local",
kind="StatefulSet", which the service map reports as a second copy of
the StatefulSet. Apply the same isKubernetesResolved guard that
NewDestinationKey got in #298.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the WithResolvedDomain method in common/net.go to ensure that already resolved Kubernetes workloads keep their own names instead of being overwritten by the FQDN. It also adds corresponding unit tests in common/net_workload_identity_test.go to verify this behavior. There are no review comments, and we have no feedback to provide.

@mayankpande88
mayankpande88 merged commit db80d39 into main Sep 18, 2026
7 checks passed
@mayankpande88
mayankpande88 deleted the fix/fqdn-overwrites-resolved-workload branch September 18, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants