Fix injected annotation removal check

[jefchien]

Description of the issue

The operator's auto-annotation flow can delete explicit opt-out annotations and then re-enable instrumentation that had previously been explicitly disabled.

For each instrumentation type, the operator pairs an insert mutation (adds the managed annotation keys with the value set to "true" when missing) with a remove mutation (deletes the same keys). Both are built from the same key/value pairs

  - instrumentation.opentelemetry.io/inject-{java,python,dotnet,nodejs}: "true"
  - cloudwatch.aws.amazon.com/auto-annotate-{java,python,dotnet,nodejs}: "true"

but the remove mutation only matches the keys when determining whether they should be deleted.

So if a pod has both managed keys and sets them to "false" to opt out, the remove mutation still sees them as operator owned and will remove them. Once removed, the opt-out is gone and the keys get repopulated with "true" on the next update re-enabling injection.

Description of change

Updated the remove mutation so it only removes annotations the operator actually added by checking the values as well as the keys. It removes the managed annotation pair only when the current values still match the injected values, so an explicit "false" opt-out is preserved.

Testing

Added a test case to the existing unit tests to assert that a value set to "false" survives when the managed value is "true".

Built the image and installed it on an EKS cluster using the amazon-cloudwatch-observability helm chart with autoMonitor.monitorAllServices: true. Used a test Deployment with a pod template that set instrumentation.opentelemetry.io/inject-java: "false" and cloudwatch.aws.amazon.com/auto-annotate-java: "false".

Reproduced the issue using the current operator 3.7.0. On create, the webhook deleted both of the "false" opt-outs. After adding a Service that points to the workload and triggering an update, the managed keys for all 4 supported languages get added including the Java ones.

With the fixed operator, the java opt-outs are preserved on create and after the update (only the 3 other language managed annotations got added). Verified that removing the Service still resulted in all the operator's managed keys (the other 3 languages) getting removed and leaving the opt-outs.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[Nebulazhang]

We reproduced this issue in a GitOps-managed EKS environment, and it creates a second failure mode beyond unintentionally re-enabling instrumentation: permanent reconciliation drift between controllers.

Environment and reproduction

• Amazon CloudWatch Observability controller image: cloudwatch-agent-operator:3.3.2

• Application Signals auto-monitoring enabled

• An Argo CD-managed Deployment with no matching Kubernetes Service

• The Pod template explicitly declares both annotations for each language as "false", for example:

  cloudwatch.aws.amazon.com/auto-annotate-nodejs: "false"
  instrumentation.opentelemetry.io/inject-nodejs: "false"
  

The stored Deployment consistently loses all eight CloudWatch/OpenTelemetry opt-out annotations. A server-side dry-run update confirmed that the annotations are removed during API admission, before persistence. Applying the same annotations to a Deployment selected by a Service preserves them.

The observable result is an Argo CD Application that remains OutOfSync / Healthy: Argo CD continuously restores the declared annotations, while the CloudWatch workload webhook continuously removes them.

Multi-controller ownership gap

This exposes a broader interoperability problem for GitOps controllers, policy operators, and admission webhooks. The current removal mutation infers ownership from key presence alone. It has no way to distinguish:

1. annotations previously generated by auto-monitor and now eligible for cleanup, from
2. annotations explicitly owned by another declarative controller or by the workload author.

Ignoring an unselected workload would avoid the conflict, but cleanup is also necessary when auto-monitor previously injected "true" and the workload is later deselected. The important distinction is therefore not selected versus unselected; it is operator-generated state versus user-declared state.

The value-aware removal in this PR is a practical compatibility boundary: auto-monitor only injects "true", so annotations whose values differ must not be treated as auto-monitor-owned cleanup candidates. It preserves explicit "false" opt-outs, prevents cross-controller reconciliation loops, and still removes the exact "true" pairs generated by auto-monitor.

An explicit ownership marker would be stronger in the long term, but would require migration semantics for annotations created by older operator versions. The proposed value check fixes the demonstrated destructive behavior without adding that compatibility surface.

This PR matches our reproduction and would allow us to remove a workload-specific Helm rendering workaround after the fixed operator reaches the EKS add-on version we consume.

[Nebulazhang]

Nice, LGTM for my issues.