anthropic: fix cursor capturing first-page ID on multi-page forward runs

[P1llus]

On subsequent runs the CEL input uses before_id paging, walking forward from the last bookmark toward newer events. The newest events appear on the last page of the run, but run_first_id was captured from the first page only. When has_more became false the persisted first_id was set from run_first_id, so the cursor advanced by only one page's worth of activity per run regardless of how many pages were fetched.

Fix the has_more=false branch: when cursor.first_id already has a value (a subsequent run), take body.first_id from the terminal page directly rather than the stale run_first_id.

Extend the system test mock to cover multi-page subsequent runs: Round 2 now spans two pages so that steady state is reached only if the cursor advances to the terminal page's event.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

The system mock now properly tests the pagination as it is intended, so the sorting order is different on page 1 and 2

elastic-package test system 
--- Test results for package: anthropic - START ---
╭───────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├───────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ anthropic │ audit       │ system    │ default   │ PASS   │ 51.082320815s │
╰───────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: anthropic - END   ---
Done

[Copilot]

🟡 Changes recommended

The PR description claims a fingerprint processor was added, but no fingerprint processor configuration exists in the anthropic package, so the description and code need to be reconciled.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes cursor advancement for the Anthropic audit CEL input when a subsequent (forward tail) run spans multiple pages, ensuring the persisted bookmark advances to the terminal page’s first_id rather than the first page’s first_id.

Changes:

• Update the has_more=false cursor persistence logic to use the terminal page’s body.first_id when cursor.first_id is already set (subsequent runs).
• Extend the system test mock to simulate a multi-page subsequent run and update the expected hit count accordingly.

File summaries

File	Description
packages/anthropic/data_stream/audit/agent/stream/cel.yml.hbs	Adjusts cursor first_id selection on the terminal page of multi-page forward runs.
packages/anthropic/data_stream/audit/_dev/test/system/test-default-config.yml	Updates the expected document count to reflect the expanded multi-page mock scenario.
packages/anthropic/_dev/deploy/docker/files/config-audit.yml	Extends the docker mock to include a two-page “Round 2” forward-tail run and steady-state validation.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[Copilot]

🟢 Approval recommended

The cursor fix is narrowly scoped, aligns with the updated system mock, and no functional issues were found beyond a minor comment clarification suggestion.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

packages/anthropic/_dev/deploy/docker/files/config-audit.yml:3

• The comment says “New events arrive newest-first”, but in this mock the newest Round 2 event (created_at 11:08) appears on page 2, not page 1. This is potentially confusing when reading/maintaining the system mock; consider updating the comment to describe the intended test invariant (newest event on terminal page) instead of “newest-first”.

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: b5ee5c2

[ilyannn]

I verified the fix empirically rather than by reading alone: extracted the CEL program and ran it with mito against this PR's elastic/stream mock config for three consecutive runs, and did the same with the pre-PR program.

Program	Run 2 cursor after the 2-page forward run	Run 3
PR	activity_01MOCKr2p2a (newest)	0 events, steady state
Base	activity_01MOCKr2p1a (first page)	re-delivers r2p2a as a duplicate

The vendor docs (Cursor-driven incremental reads) confirm the model the fix assumes: each before_id page is adjacent to the cursor and walks toward the present, so the terminal page's first_id is the value to persist. The mock extension is meaningful too, since hit_count is an exact-match assertion and the old behavior would have produced duplicates. celfmt -agent, format, lint and check all pass.

Inline: one pre-existing null-cursor stall in the branch this PR rewrites (⚠️), a mock shape nit that hides it, and a changelog wording nit.

---

These comments are drafted by 🤖 Claude Code/Fable 5.1 under my supervision.

[ilyannn]

⚠️ Null first_id on an empty page poisons the cursor and stalls collection permanently.

has(body.first_id) is true when the key is present with a null value, and state.?cursor.first_id.hasValue() is likewise true for a stored null. The vendor's own catch-up loop guards this explicitly (if page.first_id is not null: cursor = page.first_id), which implies empty pages return "first_id": null rather than omitting it.

Reproduced with mito against a mock whose empty page is {"data":[],"first_id":null,"last_id":null,"has_more":false}:

RUN 3  events=[]  cursor={"first_id":null}
RUN 4  failed eval: no such overload for format_query: type conversion error from 'null_type' to 'string'
   |   state.url.trim_right("/") + "/v1/compliance/activities?" + query.format_query()

From then on every evaluation fails before the request is built and the cursor is never rewritten, so the input is stuck until state is cleared. A brand-new tenant with no activity in the lookback window hits the same thing on its second poll (initial backfill stores {"first_id": null} via the branch below).

This is pre-existing (the base has the same has(body.first_id) test in the initial-run branch), but this PR rewrites exactly this expression and the changelog says the cursor is now correct, so it seems worth closing here. Guarding all three reads with && body.first_id != null fixes it; verified with mito against this PR's mock, a null-returning mock, and an empty-tenant mock (cursor stays on r2p2a, or stays {} and re-polls the initial window, no eval errors):

Suggested change

	(has(body.first_id) ? optional.of(body.first_id) : state.?cursor.first_id)
	: state.?cursor.run_first_id.hasValue() ?
	state.?cursor.run_first_id
	: (!state.?cursor.page_id.hasValue() && has(body.first_id)) ?
	((has(body.first_id) && body.first_id != null) ? optional.of(body.first_id) : state.?cursor.first_id)
	: state.?cursor.run_first_id.hasValue() ?
	state.?cursor.run_first_id
	: (!state.?cursor.page_id.hasValue() && has(body.first_id) && body.first_id != null) ?

The same guard belongs on the run_first_id computation in the has_more=true branch (line 88). Caveat: I could not verify the live API's empty-page shape without a compliance key; the finding rests on the documented null guard.

Minor, while here: run_first_id now only matters for the initial backfill, but the has_more=true branch still computes and carries it on every forward run. A one-line comment (or gating it on !state.?cursor.first_id.hasValue()) would save the next reader from assuming it feeds the forward path.

[ilyannn]

The steady-state body omits first_id/last_id, but per the vendor docs the API returns them as explicit nulls on an empty page. That omission is what lets the system test pass despite the null-cursor issue flagged in cel.yml.hbs. Making the mock faithful would turn this test into a regression guard for it, and hit_count: 8 stays correct:

        body: |-
          {"data":[],"first_id":null,"last_id":null,"has_more":false}

[ilyannn]

Nit: this reads like an in-run paging bug. The defect was the persisted bookmark after a multi-page forward run, which re-delivered the last page and slowed catch-up. Something like:

Suggested change

	- description: Fix incorrect pagination cursor on subsequent pages.
	- description: Fix persisted `first_id` cursor after multi-page forward runs so the bookmark advances to the newest event instead of the first page, avoiding re-delivered events.

[ilyannn]

Approving it, but note the issue above – the correctness should ideally be verified against the live endpoint, or at least the shape of the response should be clarified. Also some minor nits.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request