feat: add synchronous enrollment write strategy to prevent ghost agents

[ycombinator]

What is the problem this PR solves?

When an enrollment write is committed to fleet-server's in-memory bulk queue but the HTTP response is lost before it reaches the agent (network drop, EOF), the agent retries enrollment. If the retry lands on a different pod, that pod's FindAgent search finds nothing — the write hasn't been flushed to Elasticsearch yet — and creates a second agent document. These duplicate documents ("ghost agents") accumulate and do not resolve on their own.

The batched pre-refresh dedup approach in #7662 cannot prevent this: cross-pod retries are never in the same flush batch, so in-batch dedup never fires. Evidence from the 30k serverless checkin scale test: 35 ghost agents with enrolled_at timestamps clustered in a 21-second window (11:53:42–11:54:03 UTC), zero ErrEnrollDuplicate 429s — confirming in-batch dedup never triggered for any of the 35 retries.

How does this PR solve the problem?

A new feature flag, inputs[].server.feature_flags._sync_enrollment_write (default: false), selects the enrollment write strategy.

When false (default), the existing async bulk queue path is unchanged.

When true, createFleetAgent writes the agent document directly to Elasticsearch using op_type=create&refresh=wait_for. ES blocks until the document is committed and visible to search on all shards before returning. The enrollment response is not sent until this write completes, so any retry on any pod finds the existing document immediately. A 409 from op_type=create is treated as success.

The _ prefix on the flag name signals that it is internal and temporary. The intent is to enable it in Staging via serverless-gitops, validate with the 30k checkin scale test, and then remove the flag entirely — making the sync path the permanent enrollment write path.

How to test this PR locally

Enable the flag and run the 30k Fleet Server checkin scale test against Staging. Confirm ghost agent count is 0 after enrollment completes.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Relates feat: batch enrollment FindAgent searches with pre-refresh dedup #7662
• Relates elastic/ingest-dev#9167

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[Copilot]

Pull request overview

This PR introduces an opt-in enrollment write strategy to prevent “ghost agent” duplicates when enrollment responses are lost and the agent retries against a different fleet-server instance before the original write becomes searchable.

Changes:

• Added inputs[].server.feature_flags.sync_enrollment_write configuration flag (default false) to select enrollment write strategy.
• Implemented synchronous enrollment agent-document creation using op_type=create with refresh=wait_for, treating HTTP 409 conflicts as success.
• Added a changelog fragment describing the new feature flag and behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
internal/pkg/config/input.go	Adds the SyncEnrollmentWrite feature flag to Fleet Server’s feature flags config.
internal/pkg/api/handleEnroll.go	Plumbs the feature flag into enrollment and adds a synchronous ES write path for agent document creation.
changelog/fragments/1787924366-sync-enrollment-write.yaml	Documents the enhancement and how the new feature flag changes enrollment behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

TL;DR

Buildkite failed in Run check-ci because a test callsite wasn’t updated after createFleetAgent gained a new syncWrite bool parameter. Update handleEnroll_test.go to pass the fifth argument.

Remediation

• In internal/pkg/api/handleEnroll_test.go (around line 603), change the createFleetAgent(...) invocation to include the new boolean argument (e.g. false for existing async-path test behavior, or true if explicitly validating the sync path).
• Re-run CI (.buildkite/scripts/check_ci.sh) after the test compile fix.

Investigation details

Root Cause

createFleetAgent was changed from:

• createFleetAgent(ctx, bulker, id, agent)

to:

• createFleetAgent(ctx, bulker, id, agent, syncWrite bool)

in internal/pkg/api/handleEnroll.go (function definition near line 653 and callsite near line 425), but one test callsite still uses the old 4-argument signature:

• internal/pkg/api/handleEnroll_test.go:603

This causes go fix (run by check-ci) to fail at compile time for internal/pkg/api.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16511
• Job/step: Run check-ci (.buildkite/scripts/check_ci.sh)
• Key log excerpt:

fix: internal/pkg/api/handleEnroll_test.go:603:77: not enough arguments in call to createFleetAgent
have (context.Context, *"github.com/elastic/fleet-server/v7/internal/pkg/testing".MockBulk, string, "github.com/elastic/fleet-server/v7/internal/pkg/model".Agent)
want (context.Context, "github.com/elastic/fleet-server/v7/internal/pkg/bulk".Bulk, string, "github.com/elastic/fleet-server/v7/internal/pkg/model".Agent, bool)
Error: running "go fix ./..." failed with exit code 1

Verification

• Not run locally in this environment; diagnosis is based on Buildkite compiler output and PR diff.

Follow-up

• If this test is intended to cover the new sync enrollment strategy, consider adding/adjusting assertions for both syncWrite=false and syncWrite=true paths so future signature/behavior changes are caught explicitly.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

internal/pkg/config/input.go:131

• The PR description and example config use inputs[].server.feature_flags.sync_enrollment_write, but the code/config tag added here expects feature_flags._sync_enrollment_write. As-written, following the PR description won’t enable the feature flag, which is a discrepancy with the stated usage.

		// SyncEnrollmentWrite (config key: feature_flags._sync_enrollment_write) selects the enrollment write strategy.
		// When false (default), agent documents are written via the async bulk queue — existing behaviour.
		// When true, agent documents are written synchronously with refresh=wait_for, making the document
		// immediately searchable before the enrollment response is sent and eliminating ghost agents.
		SyncEnrollmentWrite bool `config:"_sync_enrollment_write"`

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

internal/pkg/api/handleEnroll_test.go:632

• This test checks that non-409 errors are surfaced, but it doesn’t verify that the sync-write request is using op_type=create and refresh=wait_for. Adding assertions on the outgoing request makes the test protect the behavior the feature flag is meant to enforce.

	mt.RoundTripFn = func(req *http.Request) (*http.Response, error) {

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

internal/pkg/api/handleEnroll_test.go:621

• The new sync-write enrollment tests depend on MockTransport, which is defined in handleUpload_test.go (same package). This creates a brittle coupling between unrelated tests; if the upload tests are refactored/removed, these enroll tests will stop compiling. Consider moving MockTransport to a dedicated shared test helper file (e.g. internal/pkg/api/mock_transport_test.go) or defining a local transport helper in this file.

func TestCreateFleetAgentSyncWrite409Succeeds(t *testing.T) {
	mt := &MockTransport{}
	mt.RoundTripFn = func(req *http.Request) (*http.Response, error) {
		assertSyncEnrollParams(t, req)
		return &http.Response{

[blakerouse]

Looks good.

[github-actions]

@Mergifyio backport 9.5 9.4 8.19

[mergify]

backport 9.5 9.4 8.19

✅ Backports have been created

Details

• #7720 [9.5](backport #7712) feat: add synchronous enrollment write strategy to prevent ghost agents has been created for branch 9.5 but encountered conflicts

Cherry-pick of d6324c7 has failed:

On branch mergify/bp/9.5/pr-7712
Your branch is up to date with 'origin/9.5'.

You are currently cherry-picking commit d6324c7.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/api/handleEnroll.go
	modified:   internal/pkg/config/input.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/api/handleEnroll_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #7721 [9.4](backport #7712) feat: add synchronous enrollment write strategy to prevent ghost agents has been created for branch 9.4
• #7722 [8.19](backport #7712) feat: add synchronous enrollment write strategy to prevent ghost agents has been created for branch 8.19 but encountered conflicts

Cherry-pick of d6324c7 has failed:

On branch mergify/bp/8.19/pr-7712
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit d6324c7.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/api/handleEnroll.go
	modified:   internal/pkg/api/handleEnroll_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/config/input.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally