Skip to content

shared-mesh-protocol.md

Latest commit

 

History

History
135 lines (94 loc) · 7.51 KB

shared-mesh-protocol.md

File metadata and controls

135 lines (94 loc) · 7.51 KB
Raw

Shared Cluster Protocol (Postgres multi-node)

← Docs index

This document is the canonical contract for Fusion multi-node operation on shared PostgreSQL.

The historical multi-leader SQLite mesh (HTTP task replication, settings gossip, strong-write quorum, offline task write queues) is retired. Nodes that share DATABASE_URL already share durable state at the database layer.

1. Goals and non-goals

Goals

  • One shared durable project + central state across multiple Fusion nodes.
  • Exclusive execution ownership per task via central claims + lease epochs.
  • Per-node worktrees, processes, and path mappings without live process migration.
  • Explicit degraded topology reads when peer HTTP health probes fail (membership visibility), without inventing divergent local task truth.

Non-goals

  • Scheduler failover (a peer does not take over another node’s live scheduler tick loop).
  • Live-process / in-memory session migration mid-task.
  • Multi-leader task writes when Postgres is unavailable (if the DB is down, nodes do not queue alternate task realities over HTTP).
  • Treating embedded Postgres as a multi-host shared backend (embedded is per-machine only).

Supported recovery model: lease handoff under OwningNodeHandoffPolicy (park, reassign-to-local, reassign-any-healthy) so a healthy node resumes from durable task state.

2. Terms

  • Node: A Fusion runtime/API process with a registered central.nodes row and local execution capacity.
  • Shared database: One Postgres cluster (schemas project, central, archive) reached via the same DATABASE_URL on every participating node.
  • Claim: Authoritative ownership row in central.task_claims keyed by (projectId, taskId).
  • Lease epoch: Monotonic fencing generation on the task row that invalidates stale owners after recovery.
  • Membership gossip: Optional peer HTTP exchange of known peers / metrics; does not carry task or settings payloads under Postgres.
  • Auth material: Provider credentials in per-machine auth.json (not in the shared DB by default); optional secure HTTP sync remains.

3. Data-class matrix (current truth)

Data class Mode Notes
Tasks, deps, steps, columns Shared Postgres Commit is cluster-visible; no HTTP task replication
Missions / agents config / workflows / audit Shared Postgres Same
Project + global settings Shared Postgres Settings HTTP push/pull between nodes is disabled (409)
Distributed task IDs Shared Postgres distributed_task_id_state / _reservations; always local allocator against shared rows
Checkout ownership Central claim + task mirror task_claims then task lease columns
Agent runtime / worktrees / live sessions Node-local Paths may differ via project_node_path_mappings
Auth credentials (auth.json) Node-local + optional sync sharedState.authMaterial / auth routes only
FS blobs (.fusion/tasks/*) Node-local Metadata may be in PG; bytes on the materializing host
Topology / peer metrics Registry + probes central.nodes / peers; optional gossip + health HTTP

4. Execution ownership

Claim path

  1. AgentStore.checkoutTask → CentralClaimStore.tryClaimTask (central.task_claims).
  2. Mirror winner onto the task row (tryClaimCheckout: checkedOutBy, checkoutNodeId, checkoutRunId, checkoutLeaseRenewedAt, checkoutLeaseEpoch).
  3. Scheduler/executor on the winning node run locally; other nodes must not start a second exclusive execution lane for the same claim.

Recovery path

Only MeshLeaseManager.recoverAbandonedLease(...):

  1. Prove recoverable (owner offline/error, or lease/heartbeat stale; not active local execution).
  2. Apply handoff policy when configured.
  3. Release central claim first, then clear task lease fields and bump epoch.
  4. Requeue to todo (preserve progress when appropriate).
  5. Partial split-brain → reconcileLeaseRow on a later tick.

Run-audit: task:auto-recover-lease-*, node:lease:*, node:handoff:* as applicable.

5. Membership and HTTP mesh surfaces

Still useful under shared Postgres:

Surface Role
GET /api/mesh/state Topology snapshot for dashboard Nodes UI
POST /api/mesh/sync Peer gossip: knownPeers (+ optional authMaterial only)
POST/GET /api/mesh/task-ids/* Local allocator against shared ID tables (no remote coordinator hop)
Auth sync routes Optional credential fan-out for file-local auth
mDNS discovery Join convenience, not task SoT. _fusion._tcp advertises a Fusion-owned fusion-<nodeId8> host rather than the OS hostname; set global localNetworkDiscoveryEnabled: false to disable dashboard/serve auto-start.
Docker mesh config generator Provision managed peers

Removed / disabled:

Surface Status
POST /api/mesh/tasks/create Removed — DB is the replication plane
Task/agent/mission/audit shared-state domains Removed
Settings gossip / node settings push-pull Disabled on Postgres (409)
Remote task-ID coordinator forwarding Disabled on Postgres

6. Write queue and degraded topology (narrowed)

Historical multi-leader design used meshWriteQueue for offline task write replay and meshSharedSnapshots for last-known global task state.

Under shared Postgres:

  • Do not invent local task commits when Postgres is unavailable.
  • meshWriteQueue is limited to topology / auth retry classes (membership sync / auth material), not task or settings payloads.
  • meshSharedSnapshots support degraded membership/topology reads only; they are not a substitute board store.
  • PeerExchangeService.replayPendingWritesForNode replays only those narrow scopes.

If Postgres is down, operators fix the database; nodes do not multi-master task rows over HTTP.

7. Process lifecycle

  • fn serve / fn dashboard start one process-wide PeerExchangeService and call CentralCore.startDiscovery() after the HTTP server binds the real port.
  • InProcessRuntime is project-scoped (scheduler/executor/heartbeat) and does not start mesh services.
  • HybridExecutor remains the multi-project / multi-node orchestration path when the hybrid gate enables it.

8. Security boundary

  • Peer HTTP (sync, auth, remote isolation runtime) requires node API-key authentication when configured.
  • Never log raw secrets from auth snapshots.
  • Database credentials in DATABASE_URL must not appear in logs (redaction helpers in the Postgres connection layer).

9. Operator checklist

See the Shared Postgres multi-node runbook in docs/multi-project.md.

Short form:

  1. Same external DATABASE_URL on every node.
  2. Register nodes/projects + path mappings per host.
  3. Run engines; claims enforce exclusive execution.
  4. Expect worktrees/auth/blobs to remain node-local unless you opt into auth-sync or a future blob store.

10. Historical note

Earlier revisions of this file described protocol id fusion.shared-mesh v1 with strong/queued/append-only write classes and quorum acks for multi-leader SQLite. That contract is archived by this rewrite. Implementation remnants that still mention multi-leader envelopes are compatibility shims and must not reintroduce HTTP task replication.