Repository navigation
Add durable-ioring and win-durable-ioring: epoch-driven durability over the ring - #126
Conversation
…as a SharedHandle The engineer's deferral, recorded so it is not lost. The change would touch only the private field: D-6 keeps the handle unexposed and D-8 keeps equality refused. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Brings in #119: windows-ioring-sys removes its own SharedFile and takes win-shared-os-owned-handle's SharedHandle (D-82), plus NR-inf.1 and the 0.7.0 release. Conflicts: - windows-ioring-sys DESIGN-NOTES: kept D-80, D-81 and D-82; D-81 gains an amendment marker, its files being SharedHandles now. - windows-namespace-request-sys CHECKLIST and PLANS: kept this branch's NR-2 and main's NR-inf.1, one row each. Code this branch added on SharedFile, which main never had, moved to SharedHandle so it compiles: register_shared_files and its bookkeeping, the PushRefused doc example, and the push_refusal and shared_file_registration tests. M31.3 completed and archived: removal rather than the alias it planned, plus the half of its lifetime test nothing covered -- the hold ending when the pop retires the operation. Verified by sabotage (a leaked guard), recorded in sabotage.json. The handoff callout moves to M31.1, now the last item before DI-3.2. SH-1+.1 records the ring half done and names NR-inf.1 and IT-inf.1. durable-ioring's API.md sketch drops the ring_file bridge M31.3 made unnecessary (sketch re-checked: compiles), and its notes stop describing M31.3 as pending. Gate: windows-ioring-sys clippy clean with all features and with none, all tests and doctests pass; workspace check --all-targets --locked clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A workspace member with #![forbid(unsafe_code)] (DI-D-4) and nothing else yet. windows-ioring-sys is a path-only dependency (DI-D-16); win-shared-os-owned-handle, the contract's file type (DI-D-29), is a version-and-path dependency. COMPONENT.md is the readme. Registered in release-please, with the manifest at 0.0.0 (the win-numa-sys and win-shared-os-owned-handle precedent), and in the publish workflow. This commit is not breaking, so it would bump 0.0.0 only to 0.0.1; Release-As makes the first release 0.1.0, matching Cargo.toml, and sits here because release-please attributes a commit to a package by the paths it touches. That first release cannot publish yet, and is meant not to: cargo publish --dry-run refuses with 'all dependencies must have a version requirement', naming windows-ioring-sys, until DI-3.5 pins it. COMPONENT.md's 'planned, not built' paragraph and the plan rows' matching phrases are deleted rather than updated: they restated status. Completed item: DI-3.1: Scaffold the crate. Release-As: 0.1.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
As one item, DI-3.2 covered the whole contract, and two of its requirements sat on items numbered after it: its worked-example doctests on DI-3.3's fault injection, and the conformance oracle DI-3.7 is to be built alongside it. Now DI-3.2.1 to DI-3.2.7 -- types, trait and construction; plain I/O and the Model A front end; seals in one lineage; failures and resolution; lineages, gates and flush domains; the consumer provider; ending -- with DI-3.2 checked when the last is. Each step grows the trait by what it implements. The two dependencies are recorded as open questions for the engineer, not decided. Its constraints move to the steps they bind: DI-D-18 and the ring's completion guarantees to DI-3.2.2, the one-transition failure rule to DI-3.2.4. The M31.1 prerequisite moves to DI-3.2.2, the step that needs it, and the ring checklist's handoff callout and DI-D-18 now point there. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The engineer chose the earlier option for both: CONTRACT.md's worked examples become doctests in DI-3.2.4, on a fault seam built there and reachable from doctests, rather than waiting for DI-3.3; and the conformance oracle is folded into DI-3.2 -- built in DI-3.2.2 with the readiness harness check beside it, each later step adding its rules. DI-3.7 stays as a checked record pointing there, and DI-D-28 cites DI-3.2.2. The exchange is recorded in the session file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The shared types, dioring's identities (DioringIds) and the DurableRing trait move from API.md's sketch into the crate. The trait carries only what is implemented -- its associated types, default_lineage and lineages -- and grows with each later step; DomainCompletion exists so the provider trait can be named, and gains its answers in DI-3.2.6. Records only later steps read are marked #[expect(dead_code)], so each mark turns into a warning when its step lands. Dioring::new refuses a duplicate file key and a provider domain named twice before creating anything, then creates the ring, registers construction files through register_shared_files (D-81) and buffers, and interns flush domains. Every refusal hands back what it was given. The crate's documentation is CONTRACT.md. API.md's sketch is marked superseded by the code for what the code carries, and kept frozen as the design of what is not yet built. Findings: the kernel accepts queue sizes of zero, so ring-creation failure is tested with u32::MAX; and no input tried fails a registration at completion -- a thread handle and a zero-length buffer both registered -- so those paths are unreached by tests, recorded at the code and in sabotage.json's notCoveredHere. sabotage.json: six guards, each caught, and a control that survives. Completed item: DI-3.2.1: Types, trait and construction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…as contract `EventDelivery::new` now states what `on_completion` may rely on: it runs with the ring unlocked, so it may open a scope and submit, and two invocations may overlap with no order between their completions. Both held before only as implementation. Recorded as D-83. Re-entrancy is pinned by a_callback_may_open_a_scope_and_submit, which submits a second read from inside the first read's callback under a bound and leaks the delivery on panic, so a regression fails rather than hangs. The sabotage that holds the lock across the callback was run through the harness and caught by that test alone. Overlap is a permission and is stated, not tested. Swept "EventDelivery::ring" (the drain comment named a method that no longer exists): 1 live site, updated; 5 others are historical records of its removal (CHANGELOG, archive, D-43) and stay. Completed item: M31.1: State and pin `on_completion`'s re-entrancy and concurrency on the public surface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…1.1 is done Swept M31.1 across the tree: 2 forward-looking dioring statements updated; DI-D-28's mention, the archives and the dated design session stay as written. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
WS-D-1..5 record the crate's area, Event first, PulseEvent's omission, the invariant that makes adoption unsafe, and the dependency direction. windows-threadpool-sys gets M-T14.1 and durable-ioring's DI-3.2.2 a prerequisite on WS-1.1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…without unsafe Event owns a handle to an event object: new(ResetMode, initially_signalled), set, reset and try_clone, lending its handle and converting into OwnedHandle. Adopting an existing handle is unsafe, because the handle being an event is what a consumer such as the thread pool, which cannot wait on a mutex, may rely on (WS-D-4). PulseEvent is not offered, with Microsoft's reason in the README, which is also the crate documentation (WS-D-3). The tests cover both modes and both initial states, one waiter per set against every waiter until reset, clones as the same object both ways, cross-thread set and reset, and set and reset failing on a handle without EVENT_MODIFY_STATE, the one way to reach their error path. A harness run recorded all eight sabotages caught and the control surviving. Registered in the workspace, in release-please with the manifest at 0.0.0 (nothing released), and in the publish workflow. Release-As makes the first release 0.1.0, matching Cargo.toml; it sits on this commit because release-please attributes a commit by the paths it touches. Completed item: WS-1.1: The crate, with `Event`. Release-As: 0.1.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
From<Event> for WaitableHandle is safe because an Event is never a mutex, the one target the pool cannot wait on (WS-D-4). WaitableHandle::event keeps its signature and creates through Event, so this crate makes no event of its own and drops the Win32_Security feature that existed only for CreateEventW. The documented ways to signal or reset a pool-owned event no longer need unsafe: the two ThreadpoolWait examples signal through a kept Event clone, and WaitActivation::rearm's reset-before-re-arm example resets through the callback's clone. Swept raw SetEvent/ResetEvent/CreateEventW outside tests across the workspace: 3 sites in this crate, all updated; the rest are other crates' private events, queued as win-sync-sys WS-inf.3 in a separate commit. Tests: the conversion keeps the event it was given (CompareObjectHandles, with a negative case), keeps auto- and manual-reset behaviour, a wait built from an Event fires when a kept clone is set, and the constructor walks every reset mode and initial state. Both new sabotages were caught; since this manifest's workspace-wide run fails in a dependent crate first, each was also re-applied by hand against this crate's wait tests and caught by the test its entry names. Completed item: M-T14.1: `WaitableHandle` built on `win-sync-sys`' `Event`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… events WS-M1 finished with WS-1.1 and windows-threadpool-sys' M-T14.1. WS-inf.3 now lists every non-test raw CreateEventW/SetEvent/ResetEvent site the 2026-10-07 sweep found: it had missed the ring crate's resolver, file-enumeration's completion ring and the platform probes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…othing must change first WS-inf.2 suggested private namespaces might belong in windows-namespace-request-sys, which was written from that crate's name: it covers the file-system namespace, not the kernel object namespace, and Learn files private namespaces under Synchronization. WS-D-1 now records that adding named objects is additive and names the one open question, where an owned security-attributes type should live. Swept 'namespace' across win-sync-sys and the design sessions: 2 sites, both in win-sync-sys, both updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ign before code Implementing DI-3.2.2 showed it held two things: plain I/O with the delivery every front end shares, which DI-D-18 and DI-D-28 specify, and the Model A front end type, whose behaviour DI-2.3 decided but whose API (name, constructor, which operations it offers as &self, teardown) API.md never sketched. DI-3.2.2.1 is the first; DI-3.2.2.2 the second, with its open questions listed. Swept DI-3.2.2: the oracle note, the folded DI-3.7 note and the ring crate's M31.1 handoff now name DI-3.2.2.1; archives and the dated design session stay as written. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Found by durable-ioring's DI-3.2.2.1: IoRing's write-through flag fails with Win32 error 509 on a handle opened for cached I/O, even one opened FILE_FLAG_WRITE_THROUGH, and transfers on an unbuffered one. This crate's docs present it as a cache directive and never say so, and no test exercises it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ance oracle The trait grows by readiness, add_file(_with), write(_with), read(_with) and pop, and RegisteredBufferRing adds the registered-span forms and access to the registered bytes. Dioring hands its ring to EventDelivery once construction's registrations are claimed; the callback records each completion into dioring's own queue under dioring's own lock and sets the readiness event on every empty-to-non-empty transition (DI-D-18, DI-D-28), binding only to the callback contract the ring states (D-83). Pushes share one path: file, lineage and registration are checked before anything is reserved, every refusal -- the ring's included -- hands back the buffer and the context, and only an accepted push spends an OpId. readiness() returns a win-sync-sys Event rather than an OwnedHandle, so a front end can wait on it through a thread-pool wait without unsafe; DI-D-28 and CONTRACT.md record it. The gate setters are crate-private until DI-3.2.5 honours gates, which now says to make them public again; DI-3.5 now pins the two new path-only dependencies. durable_ioring::oracle is the conformance oracle (DI-D-24): one completion per operation, with the kind and context it was pushed with, and an explicit list of what it does not check; check_readiness is DI-D-28's harness check. Each rule is tested in both directions, and the check is shown failing for a fake that never sets the signal and one that sets it once. Found: IoRing's write-through flag fails with Win32 error 509 on any cached handle and transfers on an unbuffered one. dioring's WriteCaching rustdoc records the measurement, the caching test uses it to show the flag reaches the kernel, and the ring crate's documentation is queued as its M32.1. Swept the write-through-as-latency claim: dioring's rustdoc updated; the ring crate's five sites are M32.1's. A harness run recorded all 26 sabotages as declared, the readiness ones caught by the readiness check alone; the test binary ran 30 times without a failure. Completed item: DI-3.2.2.1: Plain I/O, and the delivery every front end shares. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The engineer's decision on the proposed name and constructor: EntryDelivery::new(ring, on_entry, env) over any D: DurableRing, an FnMut handler given a non-owning DeliveryHandle, a refusal handing back the ring and the handler, and entries queued before the handover delivered. DI-3.2.2.2 now leaves open only which operations the owner and the handle offer, and teardown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
EntryDelivery<D> owns any D: DurableRing and hands its entries, one at a time and in queue order, to an FnMut handler on thread-pool threads, waiting on the instance's readiness Event through a thread-pool wait (DI-D-31). The handler runs with the instance's lock released and gets a DeliveryHandle through which it can call back in. Entries queued before the handover are delivered: new arms the wait and then sets the signal. A failed setup hands back the instance and the handler. The two questions DI-D-31 left open are decided as DI-D-32, for the engineer to confirm. The handle mirrors the trait's consumer operations as &self methods -- add_file, write, read, their _with and registered forms, the lineage queries, and registered bytes through closures over the bytes -- and withholds pop and readiness. A closure over &mut D was rejected on the ring crate's D-43, which measured exactly that letting the instance be replaced under an armed wait. Drop and into_inner quiesce first; into_inner returns the instance with any undelivered entry still queued. The mirroring's standing cost -- each later step extends the handle -- is now stated on DI-3.2. The oracle tests' fake moved to src/fake.rs, shared with the new tests. Three sabotages -- no handover signal, a wake that stops after one entry, the handler run under the instance's lock -- were each caught by the test their entry names; the last fails at its bound because the test leaks the front end rather than hang its drop. The test binary ran 30 times without a failure. Completed item: DI-3.2.2.2: The Model A front end. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
make_durable_through, durable_through, sealed_through and epoch_state, in the default lineage, through the built-in default provider (DI-D-33). One lineage's durability is an I/O-free state machine under dioring's lock; the delivery callback records each completion there and pushes the flushes it makes due while still holding it, which the ring crate's D-83 permits. A seal waits for its writes in flight, flushes each file once, and Durable is appended per seal as the high-water mark passes it, after the completions it covers. Late writes are refused as Sealed; asking again answers AlreadySealed with the epoch's state. A failed flush and a provider-routed write leave their seal pending until DI-3.2.4 and DI-3.2.6. DeliveryHandle mirrors the four new operations (DI-D-32). The conformance oracle gains the seal rules: no write accepted at or below its seal, Durable only for what was sealed, and only after every covered completion -- each tested in both directions. Thirteen new sabotages, each caught; the relay's existing ones re-anchored after its rewrite. The late- write sabotage first crashed the process (a debug assertion unwinding through an unsubmitted batch), so the check now also runs before the ring holds the push. Also removes a stray line DI-3.2.2.2's stub left behind. Completed item: DI-3.2.3: Seals and durability in one lineage, through the built-in default provider Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A failed flush is observed as a failure whose suspect set is every held write its reach covers, in push order, frozen at observation (DI-D-12, DI-D-34). Reach follows the declared flush domains: intersection for a flush, and ImportScope for an import, both through one transition with the cause as data. The trait gains resolve, import_failure, failures and take_token, mirrored on DeliveryHandle; FailureToken gains liveness, close, and a drop that closes. Heal takes effect at the first seal after it that succeeds, abandon at once; Blocked answers a seal a failure holds. An epoch passes the mark only when every failure containing it is resolved. A fault seam, Dioring::fail_next_flush behind a fault-injection feature, builds on windows-ioring-sys' own: it transforms the flush's real completion. A self dev-dependency enables it for every test build, and CONTRACT.md's worked examples are now compiled doctests driven by it; the many-to-many example's second ending is corrected to the contract's own rule. The oracle gains the failure rules, each tested both ways. Sixteen new sabotages, all caught; the manifest runs --no-fail-fast so the doctests run under each, and three sabotages fail them. Two older sabotages were caught only by a process abort on the delivery thread, so the core now records an inconsistency and pop asserts on it. Completed item: DI-3.2.4: Failures and their resolution Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The engineer's decision, DI-D-35: once an epoch is abandoned, a write tagged with it is refused, open or sealed, as PushRefusal::EpochAbandoned, handing back its buffer and context. Sealed is checked first, so a sealed abandoned epoch keeps that answer. The oracle gains the rule, applied once the Abandoned entry is observed. CONTRACT.md's guarantee 6 and resolution section, and API.md's refusal list and sketch, state it. Two sabotages, each caught by the test named for it. Completed item: DI-3.2.4.1: A write tagged with an abandoned epoch is refused Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nd queue DI-3.2.4.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…estamp decisions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A Windows-only workspace member, registered with release-please and the publish workflow as win-sync-sys is, with its README as the crate documentation. No dependencies yet: the Win32 bindings arrive with the clocks that call them. clippy::undocumented_unsafe_blocks is denied, so an unsafe block without its safety argument does not build. chore, because a crate with no API has nothing to release; WT-1.2 is the first that does. Completed item: WT-1.1: Scaffold the crate Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ock trait The two layers of WT-D-2. A Timeline is a marker type giving its ticks per second; TimePoint<T> is a u64 tick count on it that compares, subtracts and records; Ticks<T> is a signed distance on it that converts to and from Duration, truncating towards zero; and Clock reads now on a timeline, taking &self so a clock can carry state (WT-D-7, for the engineer to confirm). Points on two timelines are different types, shown by a compile_fail doctest pinned to E0308. Nine sabotages, each caught by the test named for it, and a control that survives. Completed item: WT-1.2: The two layers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion, with IoRing::into_owned IoRing<T, X, K> has a kind, Open or Owned, defaulted to Open so every existing IoRing<T, X> is unchanged. The registrations, the _raw flush and cancel and push_raw queue an operation the ring holds nothing for; they require K: AllowsRaw, which only Open has, so they do not exist for an Owned ring. A pop on an Owned ring returns (completion, payload, sidecar) with no case of the ring having held nothing. into_owned consumes an Open ring and hands it back while any such operation is in flight; there is no way back. CompletionWake and the responder registrations take either kind. D-91. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pletion it held nothing for is not a case WindowsBackend seals its ring with into_owned once its registrations are made and waited on, and pop takes the flat (completion, buffer, sidecar). The branch that returned an IoFault for a completion for an operation the backend did not push is gone: that was an inconsistency in this crate's own bookkeeping, which DI-D-59 says is not for normal error reporting, and by construction it can no longer be written. A registration left in flight at the seal panics. Needs windows-ioring-sys' typed ring kinds (D-91). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…hive DI-7+.3 Completed item: DI-7+.3: The Windows backend's ring is an owned one: its pop has no case of a completion the ring held nothing for, and the panic and error rule is recorded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ot hold, reports one it cannot grow for, and keeps its registered buffers A write ending past isize::MAX is refused as malformed at check and push. A write the target cannot find memory for completes with a fault and its buffer and is not aborted on. with_registered panics when an operation is queued or a completion is waiting, since replacing the buffers invalidates a region one may name. Review of PR 126. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…s pop fails The deliverer collapsed a failed pop into an empty queue, losing the only report of a doorbell that failed or a ring that could not be read: the wake that took the readiness edge is the only thing that told the consumer. The handler now receives a DeliveryEvent, an entry or a failure, told once per wake; delivery goes on. Review of PR 126, DI-D-58. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Completed item: DI-7+.4: Model A's handler is told when the instance's pop fails, and the in-memory backend refuses what it cannot hold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Response to the latest review round. The three findings listed as open (detach, drain failures, retrying failed submissions) are the threads already answered and resolved in Previously missed: registered buffers replaced while operations are outstanding ( Previously missed: a write whose end exceeds A thread this round raised on |
…nstead of limiting scheduling latency The interval tests required the clock's interval to be within 25 ms of Instant's, which fails whenever the scheduler pauses the thread that long between the two readings. The clock's interval now sits between an Instant interval inside it and one around it, which holds whatever the scheduler does; the interrupt and system clocks keep a 2 ms slack for reading a different counter. The same pattern was in all three. Review of PR 126. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rocess's queue_user_apc is safe and the routine is an address and a boxed closure in this process, but it took any AsHandle: a thread handle of another process with THREAD_SET_CONTEXT would have run whatever is at that address there. It now compares GetProcessIdOfThread with GetCurrentProcessId before queuing, InvalidInput for another process and the call's own error for a handle that is not a thread or lacks THREAD_QUERY_LIMITED_INFORMATION. Recorded as WS-D-9. Tested against a suspended child process's main thread; the sabotage that removes the check is caught. Swept queue_user_apc: one definition, no other callers in the workspace. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nstance did not mint LineageHandle::detached and mint are public, so a caller can make a handle for a real lineage of an instance. It has a reference count of its own, so end_lineage and retire_lineage accepted it as the last handle and abandoned epochs while the genuine handles were live, and the genuine last release then found the lineage gone and panicked. The driver now compares the home a handle reports to with its own hub, through the new LineageHandle::is_homed_at, and refuses any other as Foreign, handing it back. Recorded in DI-D-40. Tested with both forgeries (detached, and minted for a home of the caller's) through end, retire and the default lineage; the sabotage that goes back to the instance name is caught. Swept LineageHandle::detached and ::mint: the driver, the fake and ids.rs; the fake keeps nothing to end and is unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-9.8.2 done Completed item: DI-9.8.2: read the portable job's first CI run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>


Draft, opened so the portable CI job (
ci.ymlruns only on pull requests and pushes tomain) has a first run to read; this isDI-9.8.2in crates/durable-ioring/CHECKLIST.md.Adds the durable-ioring design and the
win-durable-ioringdriver overwindows-ioring-sys: epoch-driven durability intervals layered on the ring, with pool, pumped and signalled completion delivery.Two commits on this branch are breaking changes to published crates, per the ruling that a failed allocation is an error the caller receives, never a panic or abort:
windows-namespace-request-sys: security-descriptor capture andtry_clonereport allocation failure.windows-overlapped-io-sys:PageBuffers::newreturnsio::Result.Decisions live in each crate's
DESIGN-NOTES.md; the review queue is archived incrates/durable-ioring/COMPLETED-CHECKLIST.md.