spec: fix logical inconsistencies, align with fmsgd behaviour (v0.5.0) - #29
Conversation
- Verifying Message Stored also covers messages the host itself sent; previously a literal reading rejected every reply to the originator's host with code 6 (fmsgd already behaves this way: bare hash lookup in msg table, guarded by integration test 007-reply-to-own-sent). - Exactly one header-response code is sent, evaluated in precedence order (batch duplicate -> 10, add-to recipients here -> 65, add-to none here -> 11, all-recipient duplicate -> 10, else 64); the old step wording could emit contradictory sequences after a 65. - Code 11 rewritten: it acknowledges an add-to batch by a host with no add-to recipients; the "additional recipients have been accepted" wording described a state no path produces (65 covers that case). - add to addresses MUST NOT overlap to (matches fmsgd's reject), which also removes the per-recipient code-count ambiguity NOTE II created. - Add-to copies: exception list now includes has-pid flag and topic omission (topic is mutually exclusive with pid). - Unified unsupported-first-byte handling to REJECT 2 + close in both dispatch sections (matches fmsgd readVersionOrChallenge). - Outgoing-record creation is MUST in both places it is described. - message hash consistently defined via Computing Message Hash. - Numbering fixes: Sending steps 1-10, Handling a Challenge 1-5, 1.4.6.3.3 cross-ref; TOC gains Notes on Adding Recipients and Computing Message Hash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h hash Added recipients participate in (and reply to) the add-to batch message only, not the original; batches form sibling branches under the original so the thread evolves as a tree; batch identity is the batch message hash (covering time), so re-issuing the same addresses at a new time is a distinct batch, not a duplicate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Generalise Verifying Message Stored NOTE I beyond code 11: an add-to message's data never crosses the wire when the receiving host already holds the original (codes 11 and 65), and its sender already holds it, so every host holding a batch MUST be able to reconstruct the batch message (header as transmitted + original's data) to verify replies referencing the batch by its hash. The 65 path now carries the same record-the-batch-fields obligation the 11 path had, and 'sent by the host' explicitly includes the host's own add-to batches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ders A sending host's first read on Connection 1 is always a response code, so REJECT 2 is unambiguous for an unsupported message version. A challenger's next read is exactly the 32-byte CHALLENGE-RESPONSE hash, so a code byte written into that stream is indistinguishable from the start of a hash - the listener MUST TERMINATE without responding for an unsupported challenge version (first byte > 128). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Since add to from is always in from or to, the only possible notification-only domain is from's - it arises exactly when a recipient, not the original sender, adds recipients. State this beside code 11 so readers need not derive it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Code 11 in one breath: the host already holds the rest of the message, so there is no need to send it again, and it hosts no add-to recipient, so the exchange completes. Mechanics stay in the protocol steps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An add-to batch is a full message; a recorded batch matching by hash is just the message having been received. The code table states the meaning; the distinct checks (batch hash at header time, challenge-hash all-recipient dedup) stay in the protocol steps. The parenthetical preserves the restore nuance: a host holding the message for only some of its recipients continues rather than rejecting 10. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| 1. If the value is less than 128 and a supported fmsg version, continue. | ||
| 2. If the value is greater than 128 and 256 minus the value is a supported fmsg version — this is an incoming CHALLENGE and should be processed per [Handling a Challenge](#handling-a-challenge). | ||
| 3. Otherwise Host B sends REJECT code 2 (unsupported version) on Connection 1 then closes the connection completing the message exchange. | ||
| 3. Otherwise the version is unsupported. If the value is 128 or less, the peer is a sending host whose first read on this connection is a response code, so Host B sends REJECT code 2 (unsupported version) on Connection 1 then closes the connection completing the message exchange. If the value is greater than 128, the peer issued a CHALLENGE of an unsupported version and its next read is exactly the 32-byte CHALLENGE-RESPONSE hash — a response code written into that stream would be indistinguishable from the start of a hash — so Host B MUST TERMINATE the connection without responding. |
There was a problem hiding this comment.
Should just be: Otherwise TERMINATE (unsupported version - we don't know how to respond)
| #### 4. Sending a Message | ||
|
|
||
| A Sending Host (Host A) delivers a message if and only if _from_ or _add to from_ belongs to Host A's domain. When the _has add to_ flag bit is not set, the message is sent to each unique recipient domain exactly once, regardless of how many recipients share that domain. When the _has add to_ flag bit is set, the message is sent exactly once to each unique participant domain — the domains of _from_ and of every address in _to_ and _add to_, omitting _from_'s domain when _from_ is the _add to from_ (the adder is the original sender, whose host is the Sending Host) — so that all participants of the message being added to learn of the added recipients, not only the domains hosting the new recipients. A participant domain having no address in the message's _to_ or _add to_ fields is notification-only: the message exchange completes at the single "REJECT or ACCEPT RESPONSE" code in step 6 (code 11 on success, or code 6 when Host B does not hold the parent) and no per-recipient codes are exchanged. This section describes the steps Host A performs for each domain. If multiple domains exist, Host A performs these steps independently for each domain without regard to the others. | ||
| A Sending Host (Host A) delivers a message if and only if _from_ or _add to from_ belongs to Host A's domain. When the _has add to_ flag bit is not set, the message is sent to each unique recipient domain exactly once, regardless of how many recipients share that domain. When the _has add to_ flag bit is set, the message is sent exactly once to each unique participant domain — the domains of _from_ and of every address in _to_ and _add to_, omitting _from_'s domain when _from_ is the _add to from_ (the adder is the original sender, whose host is the Sending Host) — so that all participants of the message being added to learn of the added recipients, not only the domains hosting the new recipients. A participant domain having no address in the message's _to_ or _add to_ fields is notification-only: the message exchange completes at the single "REJECT or ACCEPT RESPONSE" code in step 5 (code 11 on success, or code 6 when Host B does not hold the parent) and no per-recipient codes are exchanged. (A domain hosting recipients in _to_ but none in _add to_ likewise completes at code 11 on success.) This section describes the steps Host A performs for each domain. If multiple domains exist, Host A performs these steps independently for each domain without regard to the others. |
There was a problem hiding this comment.
Remove this text from the introdution to sending a message - the steps cover the detail, re-iterating it here makes understanding the intro harder: "A participant domain having no address in the message's to or add to fields is notification-only: the message exchange completes at the single "REJECT or ACCEPT RESPONSE" code in step 5 (code 11 on success, or code 6 when Host B does not hold the parent) and no per-recipient codes are exchanged. (A domain hosting recipients in to but none in add to likewise completes at code 11 on success.)"
| 1. If the value is less than 128 and a supported fmsg version — this is an incoming message and should be processed per [Connection and Header Exchange](#1-connection-and-header-exchange). | ||
| 2. If the value is greater than 128 and 256 minus the value is a supported fmsg version, this is a CHALLENGE we support, continue. | ||
| 3. Otherwise Host A MUST TERMINATE the connection. | ||
| 3. Otherwise the version is unsupported. If the value is 128 or less, the peer is a sending host whose first read is a response code, so Host A sends REJECT code 2 (unsupported version) then closes the connection. If the value is greater than 128, the peer issued a CHALLENGE of an unsupported version and its next read is exactly the 32-byte CHALLENGE-RESPONSE hash, so Host A MUST TERMINATE the connection without responding. |
There was a problem hiding this comment.
Reduce 3. to Otherwise TERMINATE (unsupported version)
There was a problem hiding this comment.
unsupported version code 2 is never sent on the wire, perhaps should be removed (without changing existing code numbering)
Review feedback on #29: - Both first-byte branches now simply TERMINATE on an unsupported version — we don't know how to respond, so responding at all is the wrong instinct regardless of who is listening. Replaces the split-by-peer wording in both Connection and Header Exchange and Handling a Challenge. - Code 2 (unsupported version) is consequently never sent on the wire; its row is removed from the code table. Numbering is unchanged — 2 is left unassigned, as 104 already is. - Drop the notification-only re-iteration from the Sending a Message introduction; the steps cover it, and repeating it there makes the intro harder to follow. SPEC.md distilled to match.
Follows the revised ruling in markmnl/fmsg#29: an unsupported version is one we do not know how to respond in, so both branches of readVersionOrChallenge now TERMINATE without writing anything — the message-version branch (first byte <= 128) as well as the challenge branch. Code 2 is therefore never sent on the wire. RejectCodeUnsupportedVersion and its responseCodeName case are removed, with 2 left unassigned so the numbering is unchanged; responseCodeName falls through to unknown(2) if one is ever read from a peer. Its host_test row goes with it.
| When _has add to_ is NOT set: perform the steps below for each unique recipient domain. | ||
|
|
||
| When _has add to_ IS set: perform the steps below for each unique participant domain — the domains of _from_ and of every address in _to_ and _add to_. _from_'s domain is omitted when _from_ is the _add to from_ (the adder is the original sender, whose host is Host A). Domains having no address in this message's _to_ or _add to_ are **notification-only**: the exchange completes at the single response code in step 5 (code 11 on success, or code 6 when the domain's host does not hold the parent) and never reaches step 6. | ||
| When _has add to_ IS set: perform the steps below for each unique participant domain — the domains of _from_ and of every address in _to_ and _add to_. _from_'s domain is omitted when _from_ is the _add to from_ (the adder is the original sender, whose host is Host A). A domain having no address in this message's _to_ or _add to_ is **notification-only** — only _from_'s domain can be, when a recipient rather than the sender adds recipients. |
There was a problem hiding this comment.
A domain having no address in this message's to or add to is notification-only — only from's domain can be, when a recipient rather than the sender adds recipients.
This isn't clear to me, a message where an original recipient is adding someone else on a totally new domain wouldn't be notification only.
Review feedback on #29: the residual clause in the SPEC.md sending intro read as if it restricted which domains receive an add-to, which it does not — a new domain hosting an added recipient is a full participant, not notification-only. Removing the earlier re-iteration from the intro had also left the term used at the code 11 path with no definition anywhere in SPECIFICATION.md. The intro clause is now gone from SPEC.md too (matching SPECIFICATION.md), and both documents define notification-only at the code 11 bullet where it applies: a participant domain with no address in _to_ or _add to_ at all, informed only that recipients were added.
Reverts the v0.5.0 overlap prohibition in favour of fixing the
definition it was working around. Terms already defines _recipients_ as
the SET of addresses in _to_ and _add to_; the per-recipient step
contradicted it by iterating occurrences ("in _to_ order then _add to_
order"), so an address in both lists produced one code or two depending
on which rule a host followed — desyncing the response stream.
The iteration now matches Terms: one code per DISTINCT address, in the
order the address first appears scanning _to_ then _add to_. With that
settled the overlap is unambiguous, so it is permitted again, restoring
the ability to re-serve an original recipient who lost the message.
Consequently:
- the _add to_ validation rule and binary-format row no longer forbid
addresses present in _to_;
- the add-to duplicate-except list notes _add to_ may include a _to_
address;
- the sibling-branch rule is qualified — an address in both lists was
already a participant of the original and may reply on either branch,
so only recipients added and NOT already in _to_ are restricted to the
batch branch;
- NOTE II explains the overlap and the one-code guarantee.
| - If Host B has already recorded this exact add-to batch (§11) → respond code 10 (duplicate), close. | ||
| - If any _add to_ recipient belongs to Host B's domain → respond 65 (skip data). | ||
| - Otherwise → record the add-to batch (_add to from_, _add to_, _time_) per §11, respond 11 (accept add to), close. This is the path notification-only participant domains take. | ||
| - Otherwise → record the add-to batch (_add to from_, _add to_, _time_) per §11, respond 11 (accept add to), close. This is the path taken by a participant domain hosting none of the _add to_ recipients, including a **notification-only** domain — one with no address in _to_ or _add to_ at all, being told only that recipients were added. |
There was a problem hiding this comment.
remove these little arrows use semi-colon ";" instead.
| * _topic_ is omitted if the original message had one, because _pid_ is now present and _topic_ only exists on a message without _pid_. | ||
| * _add to from_ exists and is the address of the participant in the previous message adding the additional recipients, i.e. the sender. | ||
| * _add to_ exists and is addresses of the new recipients being added. | ||
| * _add to_ exists and is addresses of the new recipients being added, none of which may already be in _to_. |
There was a problem hiding this comment.
incorrect, they may indeed be in to
| * The provided digest exactly matches the SHA-256 digest computed per [Computing Message Hash](#computing-message-hash) of a message that was previously accepted, i.e. for which the host responded with "REJECT or ACCEPT CODE" 200 (accept) to at least one recipient, OR "REJECT or ACCEPT CODE" 11 (accept add to). | ||
| * The provided digest exactly matches the SHA-256 digest computed per [Computing Message Hash](#computing-message-hash) of a message that was either: | ||
| * previously accepted by the host, i.e. for which the host responded with "REJECT or ACCEPT CODE" 200 (accept) to at least one recipient, OR "REJECT or ACCEPT CODE" 11 (accept add to); or | ||
| * sent by the host, i.e. a message whose sender (_from_, or _add to from_ when the _has add to_ flag bit is set) belongs to the host's domain and which the host has transmitted or holds for sending — including each add-to message (batch) the host sent, since a reply may reference the batch by its hash. Without this, replies to a message could never be accepted by the host of the participant that sent it. |
There was a problem hiding this comment.
Unclear what this part is saying: "Without this, replies to a message could never be accepted by the host of the participant that sent it." - what is "this" here? Overall the edit seems to be concerning sending hosts may hold the message without a response code because recipient(s) belonged to the sending host itself - such messages should also be verified as stored thereby allowing resplies to reference the message.
There was a problem hiding this comment.
I think " Without this, replies to a message could never be accepted by the host of the participant that sent it." can just be removed.
Recipients are no longer described as a set. Addresses need only be
distinct within _to_ and within _add to_; an address in both lists is a
recipient of each and receives one response code per entry, which is what
the per-recipient step already iterated and what implementations already
do. This keeps the overlap permitted while leaving the response stream
unambiguous, without redefining the iteration.
Review comments:
- The add-to "duplicate except" list no longer claims _add to_ addresses
may not already be in _to_.
- Dropped the unclear trailing sentence from Verifying Message Stored
("Without this, replies to a message could never be accepted..."); the
preceding clause already carries the point.
- SPEC.md: replaced the arrow notation with semicolons throughout (31
occurrences), splitting the three sentences where that would otherwise
have produced two semicolons in one clause.
Follows the revised ruling in markmnl/fmsg#29. Recipients are no longer treated as a set: addresses need only be distinct within _to_ and within _add to_, and an address in both lists is a recipient of each, answered once for its _to_ entry and once for its _add to_ entry. That is what fmsgd already did, so localRecipients and the sender's domainRecips go back to plain wire order and the earlier deduplication here is dropped. What remains is accepting the overlap at all: - readAddToRecipients no longer rejects an add-to address present in _to_ with code 1. Duplicates within _add to_ are still rejected. - The accepted-recipient classification keys off wire position rather than set membership: entries before numLocalTo came from _to_, the rest from _add to_. Previously an overlapping address was classified exclusively as add-to, losing its _to_ record; now each entry is recorded against the list it came from, and the msg_to insert's ON CONFLICT DO NOTHING preserves its original response code as SPEC §3's closing NOTE requires. - With the overlap check gone, readAddToRecipients no longer needs the _to_ key set, so readToRecipients stops returning it. Tests updated.
markmnl
left a comment
There was a problem hiding this comment.
Check indentation of numbering and points, e.g. bullet points in https://fmsg.org which is derived from this file are collapsed.
| | v0.3.2 | 2026-05-05 | Mark Mennell | Expanded size on message and attachments data | | ||
| | v0.4.0 | 2026-08-02 | Mark Mennell | Add-to messages delivered to all participant domains; notification-only delivery completes at code 11 | | ||
| | v0.4.1 | 2026-08-03 | Mark Mennell | Hosts must retain stored messages in full, including complete recipient lists | | ||
| | v0.5.0 | 2026-08-10 | Mark Mennell | Consistency fixes: stored messages include those the host sent; exactly one header-response code; _add to_ may overlap _to_, with one response code per recipient entry; add-to copies omit _topic_; add-to batches are sibling branches whose added recipients reply to the batch message; batch identity is the batch message hash; an unsupported version TERMINATES (code 2 retired, numbering unchanged) | |
There was a problem hiding this comment.
I feel these are all clarifications of the prior versions of the spec, perhaps we could say so , e.g.: Clarified: ...
exactly one header-response code; add to may overlap to, with one response code per recipient entry; add-to copies omit topic; add-to batches are sibling branches whose added recipients reply to the batch message; batch identity is the batch message hash; an unsupported version TERMINATES
| _"recipient"_ an address in a message's _to_ or _add to_ fields | ||
|
|
||
| _"recipients"_ the set of all addresses in a message's _to_ and _add to_ fields. | ||
| _"recipients"_ the addresses in a message's _to_ and _add to_ fields. Addresses MUST be distinct within _to_, and distinct within _add to_, but an address MAY appear in both lists — it is then a recipient of each and receives a response code for each. |
There was a problem hiding this comment.
Do we need applied logic in the Terms section?
Review comments: - Terms carried applied logic (distinctness MUSTs and the per-entry response-code rule). Those belong with the rules, and already are: validation rules 2 and 3.2 carry the distinctness, NOTE II and §3 the overlap and its per-entry codes. The entry is a plain definition again. - The v0.5.0 changelog row now reads "Clarified: ..." for the entries that pin down what earlier versions already intended, with the one genuine behavioural change — an unsupported version TERMINATES, retiring code 2 — called out separately as "Changed:".
Reverts the arrow-to-semicolon substitution made during #29 review; the arrows read better in the terse condition → outcome style SPEC.md uses throughout. The three sentences that had been split to avoid two semicolons in one clause are restored to their original single-sentence form as well. Content is otherwise untouched: applying the semicolon substitution to this file reproduces the merged version byte-for-byte.
* Terminate on unsupported challenge version instead of sending code 2 A challenger's next read after sending CHALLENGE is exactly the 32-byte CHALLENGE-RESPONSE hash, so a response code written into that stream is indistinguishable from the start of a hash. Close without responding (SPEC SS10.5); code 2 remains for unsupported MESSAGE versions, where the peer's first read is always a response code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Terminate on any unsupported version, retire code 2 Follows the revised ruling in markmnl/fmsg#29: an unsupported version is one we do not know how to respond in, so both branches of readVersionOrChallenge now TERMINATE without writing anything — the message-version branch (first byte <= 128) as well as the challenge branch. Code 2 is therefore never sent on the wire. RejectCodeUnsupportedVersion and its responseCodeName case are removed, with 2 left unassigned so the numbering is unchanged; responseCodeName falls through to unknown(2) if one is ever read from a peer. Its host_test row goes with it. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Allow to/add-to overlap, treat recipients as a set Follows the revised ruling in markmnl/fmsg#29: an address MAY appear in both _to_ and _add to_, re-serving an original recipient who lost the message. What made the overlap ambiguous was not the overlap itself but the per-recipient iteration, which walked occurrences while Terms defines _recipients_ as a set. The spec now iterates distinct addresses; this brings fmsgd in line on both sides of the wire. - readAddToRecipients no longer rejects an add-to address present in _to_ with code 1. Duplicates *within* _add to_ are still rejected. - localRecipients returns one entry per distinct address, in the order it first appears scanning _to_ then _add to_. This is what sizes the per-recipient code stream on the receive side. - The sender's domainRecips is deduplicated identically. Without this it would read one byte too many for an overlapping recipient and desync the stream — the exact failure the prohibition existed to prevent. - An accepted overlapping address is now recorded against both _to_ and the batch, instead of being classified exclusively as add-to. The msg_to insert's ON CONFLICT DO NOTHING preserves its original response code, as SPEC §3's closing NOTE requires. Note this makes fmsgd accept such a message rather than reject it; for a recipient that still holds the message fmsgd answers 103 (user duplicate), which is conformant. Actually re-serving a recipient whose copy is gone is a host policy decision, unchanged here. * One response code per recipient entry, not per distinct address Follows the revised ruling in markmnl/fmsg#29. Recipients are no longer treated as a set: addresses need only be distinct within _to_ and within _add to_, and an address in both lists is a recipient of each, answered once for its _to_ entry and once for its _add to_ entry. That is what fmsgd already did, so localRecipients and the sender's domainRecips go back to plain wire order and the earlier deduplication here is dropped. What remains is accepting the overlap at all: - readAddToRecipients no longer rejects an add-to address present in _to_ with code 1. Duplicates within _add to_ are still rejected. - The accepted-recipient classification keys off wire position rather than set membership: entries before numLocalTo came from _to_, the rest from _add to_. Previously an overlapping address was classified exclusively as add-to, losing its _to_ record; now each entry is recorded against the list it came from, and the msg_to insert's ON CONFLICT DO NOTHING preserves its original response code as SPEC §3's closing NOTE requires. - With the overlap check gone, readAddToRecipients no longer needs the _to_ key set, so readToRecipients stops returning it. Tests updated.
Analysis of SPECIFICATION.md v0.4.1 found several logical inconsistencies; fmsgd was then audited to see why integration tests pass regardless, and the spec is updated to encode the working behaviour (per discussion). SPEC.md is re-distilled to match; a companion PR syncs it into fmsgd.
Major fixes
Verifying Message Stored excluded messages the host sent. The definition required a prior accept response (200/11), which a host never gives for messages it originated — a literal reading rejects every reply arriving at the originator's host with code 6, and makes the v0.4.0 add-to notification to the original sender's domain impossible. fmsgd already treats sent messages as stored (bare
SELECT id FROM msg WHERE sha256 = $1instore.go, hash populated byensureSharedHashbefore first delivery; guarded by integration test007-reply-to-own-sent.sh). The definition now includes messages the host sent.Code 11 semantics. The code table, §Reject-or-Accept and the Sending step read "the additional recipients have been accepted" — a state no execution path produces: when any add-to recipient belongs to the receiving host it responds 65, and 11 only when none do. All three places now describe code 11 as acknowledging/recording an add-to batch by a domain hosting none of the added recipients (fmsgd:
resolvePostChallengeCode,handleAddToPath).Single header-response code. §3's steps could literally emit a REJECT 10 or a 64 after a 65 had been sent — bytes the sender state machine would mis-parse as per-recipient codes. §3 step 1 now states exactly one code is determined and sent, first matching rule wins (matching fmsgd's collapse in
resolvePostChallengeCode).add tomay overlapto; one response code per recipient entry. NOTE II allowed the overlap while Terms called "recipients" a set and §3 iterated occurrences, so an address in both lists produced one code or two depending on which rule a host followed — desyncing the response stream if hosts disagreed. Terms no longer calls recipients a set: addresses need only be distinct withintoand withinadd to, and an address in both is a recipient of each, receiving one code for itstoentry and one for itsadd toentry. §3 states this explicitly.Two earlier revisions of this PR resolved the contradiction differently — first by forbidding the overlap, then by making the iteration set-based. Both were rejected in review: the overlap is useful (it re-serves an original recipient who lost the message) and per-entry codes are what implementations already do. Consequently the
add tovalidation rule and binary-format row no longer excludetoaddresses, the add-to "duplicate except" list notes the overlap, and the sibling-branch rule is qualified — an address in both lists was already a participant of the original and may reply on either branch, so only recipients added and not already intoare confined to the batch branch.fmsgd is brought in line by Allow to/add-to overlap, one response code per recipient entry fmsgd#41 (it previously rejected the overlap with code 1).
Add-to "exact duplicate except…" list omitted
topic. An add-to copy of a thread-starting message gains apid, andtopicis mutually exclusive withpid— so a conforming "exact duplicate" was structurally invalid. The exception list now includes the has-pid flag and topic omission (matching fmsgd'saddToHeader).Minor fixes
Most of the above are clarifications — they pin down what earlier versions already intended and what fmsgd already does — and the changelog row labels them as such. Bumped to v0.5.0 for the one genuine behavioural change: an unsupported version now TERMINATES rather than responding, retiring code 2 from the table (existing numbering unchanged).
🤖 Generated with Claude Code