[8.19](backport #7712) feat: add synchronous enrollment write strategy to prevent ghost agents

[mergify]

What is the problem this PR solves?

When an enrollment write is committed to fleet-server's in-memory bulk queue but the HTTP response is lost before it reaches the agent (network drop, EOF), the agent retries enrollment. If the retry lands on a different pod, that pod's FindAgent search finds nothing — the write hasn't been flushed to Elasticsearch yet — and creates a second agent document. These duplicate documents ("ghost agents") accumulate and do not resolve on their own.

The batched pre-refresh dedup approach in #7662 cannot prevent this: cross-pod retries are never in the same flush batch, so in-batch dedup never fires. Evidence from the 30k serverless checkin scale test: 35 ghost agents with enrolled_at timestamps clustered in a 21-second window (11:53:42–11:54:03 UTC), zero ErrEnrollDuplicate 429s — confirming in-batch dedup never triggered for any of the 35 retries.

How does this PR solve the problem?

A new feature flag, inputs[].server.feature_flags._sync_enrollment_write (default: false), selects the enrollment write strategy.

When false (default), the existing async bulk queue path is unchanged.

When true, createFleetAgent writes the agent document directly to Elasticsearch using op_type=create&refresh=wait_for. ES blocks until the document is committed and visible to search on all shards before returning. The enrollment response is not sent until this write completes, so any retry on any pod finds the existing document immediately. A 409 from op_type=create is treated as success.

The _ prefix on the flag name signals that it is internal and temporary. The intent is to enable it in Staging via serverless-gitops, validate with the 30k checkin scale test, and then remove the flag entirely — making the sync path the permanent enrollment write path.

How to test this PR locally

Enable the flag and run the 30k Fleet Server checkin scale test against Staging. Confirm ghost agent count is 0 after enrollment completes.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Relates feat: batch enrollment FindAgent searches with pre-refresh dedup #7662
• Relates elastic/ingest-dev#9167

  ---

  This is an automatic backport of pull request feat: add synchronous enrollment write strategy to prevent ghost agents #7712 done by Mergify.

[mergify]

Cherry-pick of d6324c7 has failed:

On branch mergify/bp/8.19/pr-7712
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit d6324c7.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/api/handleEnroll.go
	modified:   internal/pkg/api/handleEnroll_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/config/input.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

TL;DR

Buildkite failed due to an unresolved merge conflict accidentally committed into internal/pkg/config/input.go, which left conflict markers in the source and broke parsing/compilation in all packaging jobs. Resolve the conflict in that file and re-run CI.

Remediation

• Remove conflict markers and correctly merge the FeatureFlags struct changes in internal/pkg/config/input.go (around the EnableExperimental / EnableOpAMP / _sync_enrollment_write additions).
• Re-run mage check:all and packaging CI (.buildkite/scripts/release_test.sh) to confirm parse/build succeeds across x86_64/aarch64 and FIPS variants.

Investigation details

Root Cause

The failure is a code bug introduced by a bad conflict resolution state in the backport branch: internal/pkg/config/input.go contains raw git conflict markers (<<<<<<<, =======, >>>>>>>) and a malformed declaration block, causing Go parser errors before compilation can proceed.

This aligns with the existing PR metadata: the PR is labeled conflicts, and the Mergify bot comment already shows both modified: internal/pkg/config/input.go during cherry-pick.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16528
• Failing jobs:
  • Run check-ci
  • Package x86_64
  • Package x86_64 FIPS
  • Package aarch64
  • Package aarch64 FIPS
• Key log excerpts:
  • internal/pkg/config/input.go:107:1: expected 'IDENT', found '<<'
  • internal/pkg/config/input.go:108:1: expected 'IDENT', found '=='
  • internal/pkg/config/input.go:144:91: illegal character U+0023 '#'
  • syntax error: unexpected <<, expected name
• Preceding PR comment confirms unresolved conflict on the same file:
  • both modified: internal/pkg/config/input.go

Verification

Not run locally in this workflow; diagnosis is based on deterministic parser errors repeated across all failed CI jobs and architectures.

Follow-up

After resolving internal/pkg/config/input.go, a single rerun should clear all five failures because they share the same parse-time root cause.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.