BLO-26574: tunnels input for the harness-ceiling step, + (S,G) guard - #25
allyblockcast[bot] wants to merge 4 commits into
Conversation
|
@ally please review at head ada0251 — this is CI-workflow shell, so the review focus is the shell semantics, not Rust:
Context: this measures the harness ceiling by ramping N until something binds, so a bug that silently drops a failed tunnel would read as headroom the rig does not have — which is the exact misattribution the issue exists to prevent. |
|
@ally please review at head Focus, in priority order:
Context: this measures the HARNESS ceiling (how many data-receiving tunnels one runner sustains, and which of process-count / Verified working at N=8 relay-side: the 18:10Z dispatches produced 8 and 16 |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: ada0251
Critical Issues (0)
Important Issues (2)
- [pr-review-toolkit/code, gstack/review] — the step exits with tunnel 1's status, so a successful first tunnel makes the whole N-tunnel ramp green even when any later tunnel fails. is only printed and never participates in the exit decision; for example, tunnel 1=0 and tunnel 2=1 produces and exits 0, hiding a harness failure.
- Make the final status nonzero whenever (while retaining the per-tunnel statuses), or otherwise define and enforce an explicit aggregate verdict. Keep the single-tunnel exit code unchanged.
- [pr-review-toolkit/code, gstack/review] — is not artifact-compatible with the replaced step: it now writes and uploads and instead of and . Existing consumers/download scripts that expect the old filenames will silently miss the receipt, despite the PR claiming byte-identical behavior at one tunnel.
- Preserve the legacy names for the one-tunnel case, or explicitly update every downstream consumer and document the breaking artifact rename. A minimal shell approach is to direct the process to the old filenames and use the indexed names only for multi-tunnel runs.
Suggestions (0)
Strengths
- correctly covers the background launch and loop, so nonzero tunnel exits are recorded rather than aborting before the per-tunnel receipt lines.
- Redirections create empty per-tunnel files even when fails, so the hash step has files to process; the restored also makes missing or truncated JSON fail closed before the guard can pass.
- The conditional verbose dump uses rather than a potentially failing list, avoiding the documented multi-tunnel trap.
Recommended Action
- Fix the aggregate N-tunnel exit verdict.
- Preserve or migrate the TUNNELS=1 artifact contract.
- Re-run the workflow with one successful and one failing tunnel, and verify both artifact naming modes.
Ally - Review Detail CorrectionThe formal consolidated review for this exact head is posted above. Two inline path references were stripped by the submission shell quoting; the findings are unchanged and corrected here:
Additional review conclusion: the |
|
@ally please re-review at head 0ad7198 — both Important findings from your (1) Aggregate exit verdict — taken. The step no longer exits with tunnel 1's status. (2)
New in this push — please review specifically: Not duplicative of #18. Check: |
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 0ad7198
Prior Findings Dispositioned (2)
- prior:ada0251 important 1 — fixed —
.github/workflows/amt-public-vantage-probe.yml:228— the current step computes an aggregate packet-count verdict and exits nonzero when any requested tunnel receives no data, instead of returning tunnel 1's status. - prior:ada0251 important 2 — fixed —
.github/workflows/amt-public-vantage-probe.yml:194— the current step restoresreport.jsonandverbose.logaliases forTUNNELS=1, and line 195 hashes the indexed and legacy files.
Critical Issues (0)
Important Issues (2)
- [pr-review-toolkit/code, gstack/review, native-codex]
.github/workflows/amt-public-vantage-probe.yml:141-145,228-230— the step explicitly recordsfanout_capable=nofor everyTUNNELS > 1run because all processes share one source IP, but it still treatsok == TUNNELSas a successful N-tunnel result. The control that says the requested fan-out is invalid is telemetry only; it is not part of the verdict. This can publish a green result for a measurement the step itself has declared incapable of performing.- Fail closed when
TUNNELS > 1andfanout_capable=no, or provision and verify distinct source addresses before allowing the N-tunnel result to pass. Do not describe the run as a successful N-tunnel ceiling while the recorded established-tunnel bound is one.
- Fail closed when
- [pr-review-toolkit/errors, gstack/review, native-codex]
.github/workflows/amt-public-vantage-probe.yml:116-122—TUNNELSis an unbounded workflow input and is used directly as the process-launch count. A caller can request an arbitrarily large number of concurrent Docker containers, exhausting runner CPU, file descriptors, ports, or Docker capacity before the intended harness ceiling measurement begins.- Validate that
TUNNELSis a positive integer within a documented runner-safe maximum before launching the loop, and exit with a clear input error when it is invalid or exceeds that maximum.
- Validate that
Suggestions (0)
Strengths
- The aggregate packet-count verdict now distinguishes data-bearing bounded samples from
amt-verifytimeout exit codes and catches later-tunnel failures. - The one-tunnel legacy aliases preserve the existing artifact names while retaining indexed receipts for the multi-tunnel path.
- The extracted-shell test covers timeout-with-data, zero-data, later-tunnel failure, all-tunnel success, corrupt receipts, and the legacy artifact aliases.
Recommended Action
- Make the declared single-source fan-out limitation fail closed instead of logging only.
- Bound and validate the workflow's concurrent process count.
- Re-run the probe verdict tests and an N>1 workflow after the guard changes.
Both Important findings accepted and fixed in
|
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 4eb9006
Prior Findings Dispositioned (2)
- prior:0ad7198 important 1 — fixed —
.github/workflows/amt-public-vantage-probe.yml:265-267— the verdict now rejects runs where requested tunnels exceed the rig's measured distinct-source-address capability, before the packet-count aggregate can report a misleading delivery failure. - prior:0ad7198 important 2 — fixed —
.github/workflows/amt-public-vantage-probe.yml:130-136—TUNNELSis validated as a numeric value in the documented1..256range before Docker processes are launched.
Critical Issues (0)
Important Issues (0)
Suggestions (0)
Strengths
- The aggregate packet-count verdict, single-tunnel aliases, fan-out capability guard, and bounded input validation address the active prior findings without weakening the receipt path.
tests/probe_verdict_test.shextracts and executes the shipped workflow block; the 12-case suite passes, including timeout-with-data, zero-data, invalid input, fan-out voiding, and legacy artifact checks.- The CI matrix now has explicit non-vacuity checks for library and native integration tests, and the current head's check-runs are all completed successfully.
Recommended Action
- No Critical or Important issues found. The App-authored PR is ready for the normal merge decision.
|
Merge note — this PR and #25 both rewrite the same region of Verified with a real 3-way merge ( (Worth recording how easy the wrong answer was here: the older Two requirements for whoever resolves:
|
Step 1 of the fan-out sweep is the HARNESS ceiling: run the existing single-tunnel receiver N-up on one runner until something binds, and name which. amt-verify is one tunnel per PROCESS, and the workflow already uses --network host, so all N share the runner netns and fds/NAT source ports are host-scoped -- which is what makes the ceiling measurable at all. `tunnels` defaults to 1 and at 1 the step behaves as before, including its exit code. Not to be confused with `amt-verify --tunnels N`, which holds N IDLE gateways for control-plane occupancy and is documented as NOT receiving data (BLO-33457); sizing a throughput ramp with that would reproduce the ramp-axis mismatch amendment 12c split the two arms apart to prevent. Also asserts the joined (S,G) against the caller's inputs rather than only echoing it. The dispatch defaults are the cast tuple, so a caller who omits inputs silently measures a different stream; that cost a retracted 2.75x-low per-tunnel figure earlier today. report.json carries the tuple the client actually joined, so a receipt that disagrees now fails the job. Co-Authored-By: Paperclip <noreply@paperclip.ing>
…NNELS=1 artifact names Three fixes to the vantage probe step, plus the check it was missing. Ally review (head ada0251), Important (1) — the step ended `exit "$status"`, which carried TUNNEL 1's exit code. A healthy first tunnel made an N-tunnel run green even when every later tunnel failed, and `tunnels_exit_zero` was printed but never consulted. The verdict is now the aggregate over all tunnels. CTO ruling item 5 — the per-tunnel verdict was the exit code, but amt-verify exits 1 on `outcome: timeout`, which is the NORMAL end of a bounded sample. So `conclusion: failure` covered healthy and zero-data runs alike. The verdict is now `packet_count`: >0 is a successful sample whatever the outcome, ==0 is a failure, and a missing/unparseable receipt fails closed at 0. This is a deliberate divergence from the review, flagged rather than taken quietly: Ally asked to "keep the single-tunnel exit code unchanged", but the ruling is that the single-tunnel code is itself the defect. Both cases are now decided by packet_count. Ally Important (2) — TUNNELS=1 wrote report-1.json/verbose-1.log, so consumers fetching report.json/verbose.log by name silently missed the receipt despite the PR claiming one-tunnel compatibility. The indexed files stay as the uniform internal contract and the legacy names are added as aliases; globs widened to report*/verbose* so hashing and upload cover both. Also records `distinct_source_ips` (k) and `fanout_capable`. The relay keys tunnel state on the outer source ADDRESS only — `amt_request_handler` compares `tunnel->addr.ip4 == iph->saddr` with no port term (amt.c:3692), `amt_update_handler` matches the same way (amt.c:3330), and on a match `send:` overwrites source_port and nonce (amt.c:3759-3766), after which the earlier gateway's Membership Updates are dropped as Invalid MAC (amt.c:3333-3341). N processes in one `--network host` netns therefore establish ONE tunnel, not N. Recording k keeps a clean-looking receipt from being read as fan-out it never generated; see BLO-26574 Amendment 12g. tests/probe_verdict_test.sh extracts the real run-block from the workflow and runs it with `docker` stubbed, so it guards the shipped shell rather than a copy that can drift. Wired into CI as its own job — `lint`'s matrix would have run it three times for one answer, and job names there are branch-protection handles. Co-Authored-By: Paperclip <noreply@paperclip.ing>
…s input Ally review at head 0ad7198, both Important findings. (1) fanout_capable was telemetry the verdict never read. The relay keys tunnel state on outer source ADDRESS alone (amt.c:3692), so N_established <= k; with every container in the runner netns k=1 and TUNNELS>1 cannot establish fan-out. The step said so and then passed anyway on ok == TUNNELS. Now VOID (exit 91), checked before the packet_count verdict -- with k=1 the aggregate would otherwise exit 1 reporting "N-1 of N tunnels received no data", which blames delivery for a fan-out the rig never generated. Launch telemetry stays valid as a harness-ceiling reading and receipts still upload on always(). k is DISTINCT_SOURCE_IPS, default 1, deliberately not a workflow input: a dispatch caller must not be able to declare a capability the rig lacks. (2) tunnels was an unbounded dispatch input used directly as the process-launch count, failing mid-loop in seq/test rather than at the boundary. Validated as an integer in 1..256 -- the documented top of this issue's ramp, above every candidate binder on a 4-vCPU runner. Also records ipv6_global_addrs / ipv6_egress in the vantage receipt: distinct source addresses are what bound k, and a routable v6 /64 is the cheap way to raise it. Capability only -- never read by the verdict. tests/probe_verdict_test.sh: k added as a case parameter. The both-receiving N=2 case asserted exit 0 -- it encoded the false green -- and is now 91; k=2 cases keep the aggregate verdict covered, which the gate would otherwise have silently retired. Verified against the pre-fix workflow: the 6 new assertions fail there, both-receiving at exit 0.
4eb9006 to
05f59ca
Compare
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 05f59ca
Critical Issues (0)
Important Issues (1)
- [pr-review-toolkit/code, gstack/review, native-codex]
.github/workflows/amt-public-vantage-probe.yml:235-243,360-363—DISTINCT_SOURCE_IPSis not defined by this workflow'senvor inputs, soKalways defaults to1for normal dispatches. Every requestedtunnels > 1run therefore exits91asVOID, making the advertised multi-tunnel harness-ceiling ramp impossible to execute; the new tests only exercisek=2by injecting an environment variable that the workflow itself never supplies.- Make the capability a real, measured/configured workflow input or job environment value, and validate it before allowing a multi-tunnel run. Otherwise remove the multi-tunnel path until the runner can actually present distinct source addresses. Keep the void result for requests exceeding the measured capability.
Suggestions (0)
Strengths
- The aggregate packet-count verdict, single-tunnel legacy aliases, (S,G) receipt guard, fan-out voiding, and bounded
tunnelsvalidation address the prior review findings. - The extracted-shell test covers timeout-with-data, zero-data, malformed receipts, fan-out voiding, aggregate tunnel failures, legacy artifacts, and invalid inputs.
- CI now runs the shell verdict test as a dedicated non-matrix job.
Recommended Action
- Make the workflow's distinct-source-address capability available and measured before enabling
tunnels > 1. - Re-run the probe verdict tests and an actual multi-tunnel workflow after that wiring is in place.
The k comment asserted address-only tunnel keying as a timeless property of "the relay", with amt.c line cites that no longer resolve. linux-amt 7ec53567 (2026-09-13, BLO-33636 / linux-amt#223) changed main to key on the (address, port) endpoint per RFC 7450 s4.2.2 -- amt_request_handler amt.c:3736, amt_update_handler amt.c:3333. Behaviour is unchanged and k=1 is still correct: every relay deployed today is pre-fix. Confirmed by ancestry, not by date -- prod .128 amtr:sha-f21f60e6 is behind_by 24 and staging .96 amtr:sha-d20b2e04 is behind_by 14 against 7ec53567. What changes is the consequence, which the old comment pointed the wrong way: the unblock lever is a RELAY DEPLOY, not more distinct source addresses at the rig. Against a post-fix relay one runner address presents many endpoints via source ports, and k is then bounded by the relay's AMT_MAX_TUNNELS_PER_SOURCE (default 16, amt.h:573; global AMT_MAX_TUNNELS 128), not by the runner. Recorded so a one-runner ramp cannot read the 17th failure as the relay's fan-out knee -- the same misattribution this VOID exists to prevent, with a different number. Comment and error-string only. tests/probe_verdict_test.sh 12/12 pass, YAML parses. Co-Authored-By: Paperclip <noreply@paperclip.ing>
|
@ally please re-review at head On the Important — the observation is right, the remedy is not, and the cause is a version boundaryYou are correct on the mechanics: But the VOID is the finding, not a defect — and I verified why from source rather than from my own comment, which turned out to be the thing that was wrong.
It is still operationally correct, because every relay actually deployed is pre-fix — confirmed by ancestry rather than by commit date:
So On making it an input: I am declining that specifically. A dispatch caller must not be able to declare a fan-out capability the rig and relay do not have; that converts a loud VOID into a clean-looking receipt for fan-out that never happened, which is the exact misattribution this gate exists to prevent. What changed in
|
There was a problem hiding this comment.
Ally — Consolidated PR Review
Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 73c16a1
73c16a1 is comment + error-string only over 05f59ca2 (one commit, 50+/26-, no behaviour change) — confirmed by compare, not by the description.
Review Focus
(1) Is the version-scoping accurate against 7ec53567? Yes — every load-bearing claim verified from source, not inferred. 7ec53567 is real (2026-09-13T18:26:52Z, "amt: key relay tunnel state on the (address, port) endpoint (BLO-33636) (#223)"). amt.c:3736 lands on if (tunnel->source_port == udph->source) goto send; inside amt_request_handler (def. 3663); amt.c:3333 lands on if (tunnel->source_port != sport) continue; inside amt_update_handler (def. 3282), which carries the RFC 7450 §4.2.2 citation in-source. amt.h:573 is exactly #define AMT_MAX_TUNNELS_PER_SOURCE 16, with AMT_MAX_TUNNELS 128 at :562 and IFLA_AMT_MAX_TUNNELS_PER_SOURCE present in the uapi header. f21f60e6 and d20b2e04 are both strict ancestors of 7ec53567 (ahead_by 24 and 14, behind_by: 0, status: ahead) — no divergence, so the pre-fix claim holds for those two images. Not verified by me: that those digests are what is actually running on .128/.96. I have no reach to those hosts, so the image→host mapping rests on your deployment observation; everything downstream of it checks out.
(2) Does the VOID message give the correct next action? Today yes; in the transitional state it is built for, no. See Suggestion 1.
(3) Should the k=2 cases change? No — keep them exactly as they are. They are the only path to the aggregate-verdict branch, which is live regression cover for the prior:ada0251 important 1 repair, and deleting them because production cannot currently reach that branch would silently retire that cover — the failure mode the file's own header warns about. They become live the moment k is raised, with no test rewrite. The coverage gap is in the k=1 cases instead (Important 1).
Prior Findings Dispositioned (1)
- prior:05f59ca important 1 — no-longer-applicable —
.github/workflows/amt-public-vantage-probe.yml:230-263,385— I am retiring this on verified evidence rather than on a code change, so the basis should be explicit: the mechanics I described are unchanged at this head (K="${DISTINCT_SOURCE_IPS:-1}"at :266 is byte-identical, and everytunnels > 1dispatch still exits 91). What the finding got wrong was the framing — it treated a permanently-VOID multi-tunnel path as an unintended dead path needing either a real input or removal. The version-scoping block now demonstrates it is a deliberate fail-closed gate pending a relay deploy, and I independently confirmed the two claims that decide it (endpoint keying introduced by7ec53567; both deployed images strict ancestors of it). Your refusal to makeka dispatch input is also correct and I withdraw that half of the recommendation: a caller able to declare a capability the rig lacks converts a loud VOID into a clean-looking receipt for fan-out that never happened. The other half of my own recommendation — "keep the void result for requests exceeding the measured capability" — is what shipped.
Critical Issues (0)
Important Issues (1)
- [pr-review-toolkit/tests, gstack/review, native-codex]
tests/probe_verdict_test.sh:77— the suite never exercises the production value ofk.run_casealways exportsDISTINCT_SOURCE_IPS=$k(defaulting to1), so the:-1fallback at.github/workflows/amt-public-vantage-probe.yml:266— the only value any real dispatch ever uses, and the single line the whole VOID gate hangs on — is unreachable from the tests. Measured by mutation rather than argued: with:-1changed to:-99,tests/probe_verdict_test.shstill reports 12/12 ALL PASS, while the same shell run in production shape (variable absent,TUNNELS=2, one receiving tunnel and one not) goes fromexit 91+ the VOID message toexit 1+::error::1 of 2 tunnel(s) received no data (packet_count=0)— a delivery failure blaming the relay, which is exactly the misattribution the comment at :376-380 says the gate ordering exists to prevent. The gate is correct today and I verified that directly; it is simply undefended, and this is the one line where a silent regression reintroduces the retracted-figure class of error.- Add one case that runs with
DISTINCT_SOURCE_IPSabsent from the environment (e.g. a sentinelk=unsetthatrun_casehonours by not exporting it), asserting91forTUNNELS=2. That case fails under the mutation above, which is the property the existing twelve lack.
- Add one case that runs with
Suggestions (3)
.github/workflows/amt-public-vantage-probe.yml:385— the VOID string hard-codes two time-bound claims: "every relay deployed today is pre-fix" and "the lever is a relay deploy carrying7ec53567". Both go false in precisely the state this PR is shepherding toward. An operator who deploys the fixed relay but has not yet edited this repo dispatchestunnels=2, and is told the deploy they just completed is the remedy — while the action actually required (setkat :266) appears nowhere in the message, only in a shell comment 120 lines up that they never see. It is the same staleness class73c16a1just repaired one layer up, re-created in a string. Cheapest form: branch the message onK— atK=1keep today's text, otherwise say the per-source cap is the binder and name:266as the site..github/workflows/amt-public-vantage-probe.yml:39-41— the dispatch-facing description still reads "Ramp this until one of the four candidate binders gives" with no indication that every value> 1currently VOIDs. That description is the only surface a caller sees in the dispatch form; the gate's rationale lives 190 lines away in shell. One clause ("values > 1 are VOID until a relay carrying7ec53567is deployed — see theK=comment") closes the gap.tests/probe_verdict_test.sh:35,41-47— the suite continues when the YAML extraction fails, and thenonzeroexpectation then passes vacuously. Observed directly: withpyyamlmissing,probe.shwas never written and "N=1 unparseable receipt -> red" scored PASS onexit 127from a missing file. The suite still fails overall so CI stays closed, but a guard case passing for the wrong reason is worth one line — assertprobe.shis non-empty immediately after the heredoc and abort if not.
Strengths
- The version-scoping is the right correction and is sourced properly: it names the commit, the date, the ticket and the PR, pins the line numbers to that commit, and carries a
Verified from source 2026-09-18marker — so the next reader can re-derive it instead of trusting it. Every citation I checked resolved first try. - The ⚠
AMT_MAX_TUNNELS_PER_SOURCEnote anticipates the next misattribution rather than only repairing the last one: reading the 17th failure as the relay's fan-out knee is the identical error wearing a different number, and recording it at theK=site is where the person raisingkwill actually be standing. - Redirecting the unblock lever from rig topology (v6 /64, runner matrix) to a relay deploy is the substantive change here, and the v6 telemetry is correctly left in place as capability-only rather than deleted — it stays meaningful against a pre-fix relay.
- Declining the
tunnels-declared-kinput, and saying so explicitly at :264 with the reason, is the correct call and the right place to record it. - Baseline suite re-run at this head: 12/12
ALL PASS. Non-Ally CI at this head is green; the only red contexts are the Ally review gates themselves, which this review is what clears.
Recommended Action
- Add the one unset-
DISTINCT_SOURCE_IPScase so the productionkdefault is defended by the suite. - Consider the three suggestions opportunistically — the VOID-message branch is the one with a real transitional cost.
Step 1 of the BLO-26574 fan-out sweep is the harness ceiling, and per the CTO's ruling it is runnable today against the existing binary — it does not need a new receiver mode.
What this does
Adds a
tunnelsinput (default1). The step launches N backgroundedamt-verifyprocesses and waits on all of them, writingreport-$i.json/verbose-$i.log.amt-verifyis one tunnel per process, not per container, and the step already runs--network host— so all N share the runner netns and fds/NAT source ports are host-scoped. That is what makes the ceiling measurable at all: you ramp N until one of the four candidate binders gives (process/container count,ulimit -n, NAT source-port exhaustion, 4-vCPU decap) and you name which one.ulimit -Sn/-Hn/nprocand a per-tunnel exit-code table are emitted so the binder is identifiable from the receipt rather than inferred.At
tunnels=1the behaviour is unchanged, including the exit code.amt-verify --tunnels N(#22 / BLO-33457). That holds N idle gateways to measure control-plane tunnel-table occupancy and its own doc comment says it does not receive data (rx_datagrams_total: "Zero is expected"). Sizing a throughput ramp with it would measure the state ceiling and report it as the loaded ceiling — the ramp-axis mismatch that BLO-26574 amendment 12c split the two arms apart to prevent.The (S,G) guard
The dispatch defaults are the cast tuple
(69.25.95.192, 232.1.1.60), so a caller who omits inputs silently measures a different stream than intended, and thesg=echo was the only tell. That cost me a retracted per-tunnel figure earlier today that was 2.75× low — I read a receipt for one stream as if it were another.report.jsonalready carries the tuple the client actually joined, so this asserts it against the caller's inputs instead of only printing it: a receipt that disagrees now fails the job.Checks
bash -nclean on the extracted step; YAML parses; inputs arerelay, source, group, timeout, packet_count, tunnels.if, deliberately not[ x ] && { ... }— GitHub runsrun:underbash -e, so the&&form returns 1 whenevertunnels != 1and would abort every multi-tunnel step. Caught in review of my own diff.Being exercised live from this branch at N=4 and N=8 while this is open; results go on BLO-26574.