fix(spaces): gate invalidation checks on the delta and cap their server time - #228
Merged
Merged
Conversation
…er time On 2026-09-24 a routine space admin grant made AuthorityResolver's preset-deactivation check never finish on rdf4j 6.1.0. The client gave up after its 60 s socket timeout, but rdf4j kept evaluating the abandoned query, and the cycle retried it every tick: on one instance it was started 77 times over 85 minutes, none finished, and the pile-up took the fleet down. The rdf4j request logs show the same store answered the identical query in 10 s on lmdbpool4 right after the rollback. Two changes make the incremental cycle robust against this, on any rdf4j version: - Every check in applyInvalidations needs one kind of record among the new loads: an invalidator, a preset assignment, a role revocation or a role detachment. The cycle now reads which kinds the delta holds with four two-pattern ASKs (DeltaKinds) and runs a check, and the two invalidator-based deletes, only when its kind is present. Each gate is a sub-pattern of its check's own WHERE clause, so skipping is exact. Most cycles (admin grants, role assignments, new spaces) hold none of these and no longer evaluate the large joins at all. On a copy of petapico's store, all four gates return false in <= 20 ms on both 6.1.0 and lmdbpool4 for the outage cycle. - The checks, invalidation deletes and tier inserts carry a server-side time limit (setMaxExecutionTime) equal to the client's socket timeout, now the shared TripleStore.SOCKET_TIMEOUT_SECONDS. It never cuts off a request that succeeds today; it stops rdf4j from evaluating a request the client has already abandoned. An update stopped this way rolls back as a whole. InMemoryTripleStore now records the ASK and UPDATE texts production code prepares, so tests can assert which checks ran. New pipeline tests cover the gates, the skip on an admin-grant delta (only the four gates run), an applied invalidation through its gate, and a delta holding every kind, which runs all 17 operations for real; that keeps the proof that every check parses and executes, which the empty-graph test no longer gives. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tkuhn
force-pushed
the
fix/authority-resolver-query-hardening
branch
from
September 25, 2026 07:28
5c38556 to
536ad06
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On 2026-09-24 a routine space admin grant made AuthorityResolver's preset-deactivation check never finish on rdf4j 6.1.0. The client gave up after its 60 s socket timeout, but rdf4j kept evaluating the abandoned query, and the cycle retried it every tick. On petapico it was started 77 times over 85 minutes, none finished, and the pile-up took the fleet down. The rdf4j request logs show the same store answered the identical query in 10 s on lmdbpool4 right after the rollback. The 6.1.0 hang itself does not reproduce offline (a copy of that store in a fresh 6.1.0 process answers in 6 s), so this PR makes the cycle robust against it on any version rather than fixing rdf4j.
What
AuthorityResolver.DeltaKinds). Every check inapplyInvalidationsneeds one kind of record among the new loads: an invalidator, a preset assignment, a role revocation or a role detachment. The cycle now reads which kinds the delta holds with four two-pattern ASKs and runs a check (and the two invalidator-based deletes) only when its kind is present. Each gate is a sub-pattern of its check's own WHERE clause, so skipping is exact. Load counter and load data commit in one transaction, so a gate evaluated after the counter is read sees every load the check would.setMaxExecutionTimeequal to the client's socket timeout (now the sharedTripleStore.SOCKET_TIMEOUT_SECONDS, 60 s). It never cuts off a request that succeeds today; it stops rdf4j from evaluating a request the client has already abandoned. An update stopped this way rolls back as a whole, so a timed-out cycle can no longer leave half its writes behind.deployment-rationale.mdgets the log findings and the mitigation; one sentence indesign-role-revocation.mdno longer says the displacement DELETE runs every cycle.Validation
spacesstore, for the outage cycle (graph_3565, lastProcessed 3568): all four gates return false in ≤ 20 ms on both 6.1.0 and lmdbpool4, so none of the 11 checks or 2 deletes would run. Over the whole load history the gates return true where those records exist, still in ≤ 20 ms.timeoutparameter for queries and updates over HTTP; both versions aborted at the limit with a 503 in local tests.InMemoryTripleStorenow records the ASK/UPDATE texts production code prepares. New pipeline tests: gates only see records inside the delta; an admin-grant delta runs exactly the four gates; a delta holding every kind runs all 17 operations for real (keeps the proof that every check parses and executes, which the empty-graph test no longer gives); an invalidation through its gate still deletes the row and raises the rebuild flag.mvn test: 522 tests, 0 failures.Not covered
🤖 Generated with Claude Code