Skip to content

fix(spaces): gate invalidation checks on the delta and cap their server time - #228

Merged
tkuhn merged 1 commit into
mainfrom
fix/authority-resolver-query-hardening
Sep 25, 2026
Merged

tkuhn merged 1 commit into
mainfrom
fix/authority-resolver-query-hardening

Conversation

@tkuhn

@tkuhn tkuhn commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Why

On 2026-09-24 a routine space admin grant made AuthorityResolver's preset-deactivation check never finish on rdf4j 6.1.0. The client gave up after its 60 s socket timeout, but rdf4j kept evaluating the abandoned query, and the cycle retried it every tick. On petapico it was started 77 times over 85 minutes, none finished, and the pile-up took the fleet down. The rdf4j request logs show the same store answered the identical query in 10 s on lmdbpool4 right after the rollback. The 6.1.0 hang itself does not reproduce offline (a copy of that store in a fresh 6.1.0 process answers in 6 s), so this PR makes the cycle robust against it on any version rather than fixing rdf4j.

What

  • Gates (AuthorityResolver.DeltaKinds). Every check in applyInvalidations needs one kind of record among the new loads: an invalidator, a preset assignment, a role revocation or a role detachment. The cycle now reads which kinds the delta holds with four two-pattern ASKs and runs a check (and the two invalidator-based deletes) only when its kind is present. Each gate is a sub-pattern of its check's own WHERE clause, so skipping is exact. Load counter and load data commit in one transaction, so a gate evaluated after the counter is read sees every load the check would.
  • Server-side time limit. The checks, invalidation deletes and tier inserts carry setMaxExecutionTime equal to the client's socket timeout (now the shared TripleStore.SOCKET_TIMEOUT_SECONDS, 60 s). It never cuts off a request that succeeds today; it stops rdf4j from evaluating a request the client has already abandoned. An update stopped this way rolls back as a whole, so a timed-out cycle can no longer leave half its writes behind.
  • Docs: the 6.1.0 paragraph in deployment-rationale.md gets the log findings and the mitigation; one sentence in design-role-revocation.md no longer says the displacement DELETE runs every cycle.

Validation

  • On a consistent copy of petapico's spaces store, for the outage cycle (graph _3565, lastProcessed 3568): all four gates return false in ≤ 20 ms on both 6.1.0 and lmdbpool4, so none of the 11 checks or 2 deletes would run. Over the whole load history the gates return true where those records exist, still in ≤ 20 ms.
  • rdf4j honours the timeout parameter for queries and updates over HTTP; both versions aborted at the limit with a 503 in local tests.
  • InMemoryTripleStore now records the ASK/UPDATE texts production code prepares. New pipeline tests: gates only see records inside the delta; an admin-grant delta runs exactly the four gates; a delta holding every kind runs all 17 operations for real (keeps the proof that every check parses and executes, which the empty-graph test no longer gives); an invalidation through its gate still deletes the row and raises the rebuild flag.
  • mvn test: 522 tests, 0 failures.

Not covered

  • The tier inserts and the late-arrival sweep are not gated, only time-limited. The attachment-tier insert is what timed out on nanodash.net that day, so a gate there is a possible follow-up.
  • If 6.1.0's hang is in query planning rather than evaluation, the time limit may not interrupt it. The fleet stays on lmdbpool4.

🤖 Generated with Claude Code

…er time

On 2026-09-24 a routine space admin grant made AuthorityResolver's
preset-deactivation check never finish on rdf4j 6.1.0. The client gave up
after its 60 s socket timeout, but rdf4j kept evaluating the abandoned
query, and the cycle retried it every tick: on one instance it was started
77 times over 85 minutes, none finished, and the pile-up took the fleet
down. The rdf4j request logs show the same store answered the identical
query in 10 s on lmdbpool4 right after the rollback.

Two changes make the incremental cycle robust against this, on any rdf4j
version:

- Every check in applyInvalidations needs one kind of record among the new
  loads: an invalidator, a preset assignment, a role revocation or a role
  detachment. The cycle now reads which kinds the delta holds with four
  two-pattern ASKs (DeltaKinds) and runs a check, and the two
  invalidator-based deletes, only when its kind is present. Each gate is a
  sub-pattern of its check's own WHERE clause, so skipping is exact. Most
  cycles (admin grants, role assignments, new spaces) hold none of these
  and no longer evaluate the large joins at all. On a copy of petapico's
  store, all four gates return false in <= 20 ms on both 6.1.0 and
  lmdbpool4 for the outage cycle.

- The checks, invalidation deletes and tier inserts carry a server-side
  time limit (setMaxExecutionTime) equal to the client's socket timeout,
  now the shared TripleStore.SOCKET_TIMEOUT_SECONDS. It never cuts off a
  request that succeeds today; it stops rdf4j from evaluating a request
  the client has already abandoned. An update stopped this way rolls back
  as a whole.

InMemoryTripleStore now records the ASK and UPDATE texts production code
prepares, so tests can assert which checks ran. New pipeline tests cover
the gates, the skip on an admin-grant delta (only the four gates run), an
applied invalidation through its gate, and a delta holding every kind,
which runs all 17 operations for real; that keeps the proof that every
check parses and executes, which the empty-graph test no longer gives.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@tkuhn
tkuhn force-pushed the fix/authority-resolver-query-hardening branch from 5c38556 to 536ad06 Compare September 25, 2026 07:28
@tkuhn
tkuhn merged commit 9363d69 into main Sep 25, 2026
8 checks passed
@tkuhn
tkuhn deleted the fix/authority-resolver-query-hardening branch September 25, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant