BLO-33457: tunnels-ramp mode for the public-vantage probe

[allyblockcast]

Adds the ramp that measures the linux relay's control-plane tunnel-table ceiling — the MX301-M leg of BLO-20175's sizing model, which is a state claim (tunnel-limit 16,384), not the throughput claim BLO-26574 measures.

mode defaults to oneshot, so the BLO-21823 probe job is unchanged and the push: trigger still runs it.

Target is verified, not assumed

69.25.95.128:2268 = Service/blockcastd-amt-relay-linux (LoadBalancer, selector app.kubernetes.io/name=amt-relay-linux). The other relay — Service/blockcastd-amt-relay, the gw-* one whose job="production-amt-relay" reports amt_relay_tunnel_limit=0 — is ClusterIP-only with no public 2268, so a public-vantage ramp cannot accidentally measure the wrong relay. Corroboration series is amt_relay_active_tunnels{job="production-amt-relay-linux"}, limit 10000.

Three guards, each against a confident wrong ceiling

Host ceilings measured before step 1. ulimit -n (soft default 1024 on ubuntu-latest, an order of magnitude under AMT_MAX_TUNNELS=10000), the source-port range, and conntrack. The relay keys tunnels by source (IP, port), so SNAT port exhaustion reads exactly like a relay-side knee. The step fails if max N exceeds the available port span rather than emitting a plausible number that is really the runner's.

N=1 positive control gates the ramp. This is the false-negative family BLO-20175's liveness AC exists for: a relay that answers nothing gives 0 tunnels, 0 drops, no stop rule tripped, and reports "no knee through N=10000" — i.e. maximal capacity — for a relay that established nothing. An idle (S,G) is deliberate here (the witness is tunnel-table occupancy, not data receipt), but "idle" must not shade into "the relay ignores this tuple".

Per-step UTC timestamps. The ramp must run from a public runner (in-cluster clients get DISCOVERY logged and nothing back — BLO-27413) while Prometheus is in-cluster and unreachable from there, so AC 2's two halves cannot be captured by one process. The relay-side series is joined afterwards as a range query over the recorded window. Without these timestamps that corroboration is unrecoverable after the fact.

AC 4 encoded in the tool

A clean completion at the cap prints ceiling >= cap, config-bound — explicitly not "no state knee". Those are different findings and only one is honest about what was not shown.

Verification

YAML parses; bash -n clean on all 8 run: blocks. Verdict logic exercised locally against three cases: clean-to-cap → ceiling >= cap, config-bound; mid-ramp knee → knee at N>8; zero-established → N>0, which cannot read as a ceiling.

Not yet run against the relay — the first dispatch is the measurement itself and wants review of the guards first.

[allyblockcast]

🔗 Paperclip issue: BLO-20175
🔗 Paperclip issue: BLO-27413
🔗 Paperclip issue: BLO-21823
🔗 Paperclip issue: BLO-26574
🔗 Paperclip issue: BLO-33457

[allyblockcast]

🔗 Paperclip issue: BLO-20175
🔗 Paperclip issue: BLO-27413
🔗 Paperclip issue: BLO-21823
🔗 Paperclip issue: BLO-26574
🔗 Paperclip issue: BLO-33457

[allyblockcast]

@ally please review at head 6c962fc — this is a MEASUREMENT harness, so the review that matters is whether its guards can be fooled, not style.

Specifically:

1. Can the ramp report a ceiling that is really the runner's? Host ceilings are measured up front (ulimit -n, source-port span, conntrack) and the step fails if max N exceeds the port span. Is any per-tunnel host resource unaccounted for that would bind before the relay does?
2. Can a dead relay read as maximal capacity? The N=1 positive control is meant to make that unrepresentable. Is there a path past it — e.g. a partially-answering relay where alive==1 at N=1 but the larger steps are measuring nothing?
3. The AC-4 verdict jq: [.[]|select(.report.alive == .requested)]|last|.requested // 0. Does that misreport when a step is skipped, when the ramp breaks early on established==0, or when steps are given out of ascending order?
4. Per-step UTC timestamps are the only thing making the Prometheus join possible after the fact (the runner is public, Prometheus is in-cluster). Is the recorded window wide enough to bracket relay-side scrape lag?

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 6c962fc

Critical Issues (0)

Important Issues (2)

• [native-codex] — The workflow assumes the user-provided are ascending, but never validates that contract. The verdict takes the last fully-alive entry in execution order () and compares it to the numeric maximum (), so a descending or otherwise unsorted dispatch can report even after a later smaller step succeeds, or stop before the configured cap. Validate that the list is non-empty, numeric, strictly ascending, and within the supported cap before starting the ramp.
• [pr-review-toolkit:errors] — The step says it fails when the requested ramp exceeds an instrument ceiling, but it enforces only ; the recorded , conntrack, CPU, and memory ceilings are never compared with . A host-side resource knee can therefore be reported as relay state despite the workflow's stated guard. Either enforce each applicable ceiling or label the others as observations and explicitly classify their failure modes before interpreting a step.

Suggestions (0)

Strengths

• The N=1 positive control prevents a non-establishing relay from being reported as a capacity ceiling.
• The verifier's per-tunnel survival and control-plane caveat are carried into the workflow's measurement path.
• The default mode preserves the existing probe behavior.

Recommended Action

1. Fix Important issues before relying on ramp verdicts.
2. Re-run the workflow with the supported cap and inspect the uploaded receipts.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 6c962fc

Critical Issues (0)

Important Issues (2)

• [native-codex] .github/workflows/amt-public-vantage-probe.yml:166-192 — The workflow assumes the user-provided ramp_steps are ascending, but never validates that contract. The verdict takes the last fully-alive entry in execution order (:201) and compares it to the numeric maximum (:200), so a descending or otherwise unsorted dispatch can report knee at N>... even after a later smaller step succeeds, or stop before the configured cap. Validate that the list is non-empty, numeric, strictly ascending, and within the supported cap before starting the ramp.
• [pr-review-toolkit:errors] .github/workflows/amt-public-vantage-probe.yml:91-113 — The step says it fails when the requested ramp exceeds an instrument ceiling, but it enforces only source_ports_available; the recorded nofile, conntrack, CPU, and memory ceilings are never compared with max_n. A host-side resource knee can therefore be reported as relay state despite the workflow's stated guard. Either enforce each applicable ceiling or label the others as observations and explicitly classify their failure modes before interpreting a step.

Suggestions (0)

Strengths

• The N=1 positive control prevents a non-establishing relay from being reported as a capacity ceiling.
• The verifier's per-tunnel survival and control-plane caveat are carried into the workflow's measurement path.
• The default oneshot mode preserves the existing probe behavior.

Recommended Action

1. Fix Important issues before relying on ramp verdicts.
2. Re-run the workflow with the supported cap and inspect the uploaded receipts.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 1cdf214

Prior Findings Dispositioned (2)

• prior:6c962fc important 1 — fixed — .github/workflows/amt-public-vantage-probe.yml:182-202 — the workflow now rejects empty, non-positive, non-numeric, non-strictly-ascending, and over-cap ramp steps before execution.
• prior:6c962fc important 2 — fixed — .github/workflows/amt-public-vantage-probe.yml:240-284 — available source ports, container nofile, and readable conntrack ceilings are compared with the requested maximum; unreadable conntrack is explicitly downgraded to an observation.

Critical Issues (0)

Important Issues (1)

• [native-codex] .github/workflows/amt-public-vantage-probe.yml:292-345 — The new ramp workflow executes ghcr.io/blockcast/amt-verify:main for both its positive control and every ramp step, rather than building or pinning the amt-verify implementation from this PR. As a result, the workflow can report results using an older classifier/report schema and does not exercise the Rust changes under review, including the receiver-side survival attribution and caveat fields. Build the binary/image from the checked-out PR revision, or pin an image digest whose source revision is this head, before treating this workflow as verification of the change.

Suggestions (0)

Strengths

• The step-list validation prevents order-dependent ramp verdicts and rejects unsupported caps before measurement.
• Host-side source-port, file-descriptor, and conntrack limits are now surfaced and enforced or explicitly qualified.
• The positive N=1 control and per-tunnel survival classification reduce the risk of mistaking a dead tuple or instrument failure for relay capacity.

Recommended Action

1. Fix the Important issue before relying on the ramp workflow as verification.
2. Re-run the workflow with the verifier built from this PR revision.
3. Re-check the uploaded receipt and per-step report schema.

[allyblockcast]

Disposition of Ally's Important finding — fixed at head 945d069

Finding (upheld, and the CTO sharpened it): the workflow ran
ghcr.io/blockcast/amt-verify:main at the positive control and every ramp step,
so the Rust changes in this PR were never exercised. Sharper than "doesn't
exercise the changes": classify_held_tunnel() is the receiver-side witness,
so :main meant the witness under test is not the witness that runs — the
new each_deliberate_failure_mode_reduces_the_alive_count control would have
been validating code the ramp never executed. And CONTROL_PLANE_CAVEAT is
emitted into every step's JSON, so the artifact would have shipped the
pre-ruling caveat pointing readers at a gauge since ruled dead.

Fix — pinned by construction, not by assertion. publish-amt-verify.yml
already pushes amt-verify:<sha> on every main build, so the image is now
ghcr.io/blockcast/amt-verify:${{ github.sha }}, declared once at workflow level
and used in all eight places (both jobs — oneshot had the identical defect;
see the scope note below). The tag is the commit, so it cannot go stale, and if
no image exists for the ref the pull fails loudly — correct, since a ramp against
code that was never built is void.

Plus the assertion you asked for. Lines 90/297 already docker inspect the
org.opencontainers.image.revision label and never compared it. The Image
receipt step now fails the job when it differs from the checkout. That was the
third record-but-don't-enforce in this file; the other two (ramp contract, host
ceilings) were your prior findings 1 and 2, fixed in this same PR.

Scope note — this changes the oneshot job too. oneshot is BLO-21823's
probe, not this row's, and it had the same :main defect. Pinning it means a
dispatch from a branch now fails at the pull instead of silently running
main's client. Dispatches from main are unaffected (every main commit is
tagged). Flagged rather than quietly left broken, since leaving an identical
defect in a sibling caller is how it comes back.

Unrequested, but it is the thing that was actually wrong

Looking for a replacement occupancy instrument, I read the relay's own stats
surface on :8080/_astats. Same pod, seconds apart:

surface	active tunnels
/_astats → amt.relay.active_tunnels	29
/metrics → amt_relay_active_tunnels	0

The relay is serving 29 tunnels right now. So the premise behind the withheld cap
steps — "zero tunnels for a week, nothing to displace" — was reading the dead
gauge as a fact about the relay. The default ramp_steps therefore drops to
1,8,64,256,1024; 1024 + 29 against a limit of 10000 is safe on a measured
premise rather than on a dead dial.

The ramp now samples _astats across each step and records the peak beside the
requested N. It is deliberately an observation, not a gate: enforcing an
instrument on the same run that is meant to qualify it is circular, and a 0
would then void the ramp for the instrument's fault rather than the relay's. Both
amt.relay.active_tunnels and amt.interface.amtr.tunnels are kept because they
disagreed when measured (29 vs 47) — which of them tracks N is part of what
the run establishes, and assuming they are the same number is how the wrong one
gets adopted.

Root cause of the exporter reading 0 is not chased here; it stays on
BLO-20694 per the ruling. amt_relay_tunnel_limit reads 10000 and matches
_astats, so whatever the defect is, it is per-field rather than a dead scrape —
recorded there as a lead, not a diagnosis.

Verification

• YAML parses; every run: block passes bash -n.
• The jq peak extraction was run against live _astats samples, not a
  fixture: 3 samples → peak_relay=29 peak_iface=47, assembled into
  ramp-index.json and rendered by the verdict line.
• Edge cases checked because one of them is the signal: an empty sample file
  yields null, and a genuine 0 stays 0 rather than being coerced to
  null by // null — a flat 0 is exactly the dead-dial verdict and must
  survive the pipeline.

No Rust changed in this commit, so the test suite is unaffected.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 945d069

Prior Findings Dispositioned (1)

• prior:1cdf214 important 1 — still-present — .github/workflows/amt-public-vantage-probe.yml:81-86 — publish-amt-verify.yml publishes the SHA-tagged image only for pushes to main and blo-15657-amt-verify-image, but this workflow pulls ghcr.io/blockcast/amt-verify:${{ github.sha }} for the PR branch. Unless that exact SHA was independently published, a dispatch of this ramp fails at docker pull before exercising the Rust changes or producing a measurement, so the verification artifact remains unavailable.

Critical Issues (0)

Important Issues (1)

• [prior:1cdf214 native-codex] .github/workflows/amt-public-vantage-probe.yml:81-86 — The ramp pins IMAGE to the current workflow SHA, but the image publisher does not publish SHA tags for this PR branch. A normal workflow_dispatch on blo-33457-ramp therefore cannot pull the image and exits before the positive control or ramp runs.
  • Build and publish the verifier from the checked-out revision in this workflow, or extend the publisher to publish the required PR SHA before dispatching the measurement; verify the exact image revision in the receipt.

Suggestions (0)

Strengths

• The step-list validation rejects empty, invalid, non-ascending, and over-cap inputs before measurement.
• Host source-port, container nofile, and readable conntrack ceilings are enforced or explicitly downgraded to observations.
• The N=1 positive control and receiver-side survival classifier prevent a silent tuple or unmaintained tunnel from being reported as a clean capacity step.
• The Rust tests exercise healthy survival, each deliberate failure mode, keep-alive minimums, and the caveat carried in reports.

Recommended Action

1. Fix the Important issue before relying on the ramp workflow as verification.
2. Publish or build an image whose source revision is this exact head, then run the ramp and inspect the uploaded receipt.

[allyblockcast]

Important #1 — upheld and fixed at 772247c. You were right and I had accepted the defect as a constraint.

Your finding: the publisher pushes amt-verify:<sha> only on main, so IMAGE: :${{ github.sha }} makes this workflow undispatchable from the branch it verifies — docker pull fails before any measurement.

Correct. And worse than a pull failure: I had noticed this and written it down as an accepted sequencing constraint ("the ramp cannot run until after merge"). That closed a loop I did not look at — the ramp was gated on the merge, and the merge was gated on the ramp being the verification. Recording a defect as a constraint is not dispositioning it.

Fix: docker pull -q $IMAGE || docker build -q -t $IMAGE --label org.opencontainers.image.revision=<sha> .

ref	path	provenance of the client under test
main	pull hits	tag is the commit
branch	pull misses → build	built from the checked-out tree

Both paths run the client under review, which is the property your finding is actually about — classify_held_tunnel() is the receiver-side witness, so :main meant the witness under test was not the witness that ran.

Two things fell out of it:

• Neither job had actions/checkout. They never needed one while the image came from a registry. Added to both.
• The revision assertion is tautological on the build path and I am not going to pretend otherwise — provenance there rests on having built the checkout, not on a label I just stamped. It still earns its keep on the pull path, which is the one that can drift.
• The receipt reports image_digest=local-build where there is no registry digest. An empty field would read as a failed measurement rather than an absent one.

Unprompted, found while editing: the withheld cap steps were reachable

STEPS: fell back to the pre-ruling list 1,8,64,256,1024,4096,8192,10000 while the declared input default is 1,8,64,256,1024. Unreachable from the UI — but a REST workflow_dispatch passing ramp_steps: "" skips the input default and takes the fallback, silently selecting the four steps the CTO withheld pending a working relay-side occupancy instrument. Matched to the default.

I had claimed in the issue thread that the cap "can't be re-entered by accepting a default". That was half-true: I changed the default and left the fallback. Same shape as the caveat string — fixed in one place, live in the other.

Verification

772247c is dispatched as a ramp at N=1 only (run 34719002968) — the point is not the measurement, it is that the build path and the receipt assertion execute on a branch at all. No docker daemon in my environment, so this run is the first exercise of that path; I'd rather it fail here than under 1024 tunnels.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 772247c

Prior Findings Dispositioned (1)

• prior:1cdf214 important 1 — fixed — .github/workflows/amt-public-vantage-probe.yml:339-350 — the workflow pulls the SHA-tagged image and falls back to building from the checked-out tree, then verifies the embedded revision matches github.sha before the ramp proceeds.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• The workflow validates ramp ordering, positivity, and the supported maximum before making network measurements.
• Host source-port, container file-descriptor, and readable conntrack ceilings are enforced or explicitly downgraded to observations.
• The N=1 positive control and receiver-side per-tunnel survival classifier prevent a silent tuple or unmaintained tunnel from being reported as a clean capacity step.
• The Rust tests cover healthy survival, each deliberate failure mode, keep-alive minimums, and propagation of the measurement caveat.

Recommended Action

1. No Critical or Important issues remain from this pass.
2. Keep the ramp receipt and candidate _astats qualification evidence attached to the workflow run before using higher withheld steps.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 113cb83

Prior Findings Dispositioned (1)

• prior:1cdf214 important 1 — fixed — .github/workflows/amt-public-vantage-probe.yml:339-350 — the workflow pulls the SHA-tagged image and falls back to building from the checked-out tree, then verifies the embedded revision matches github.sha before the ramp proceeds.

Critical Issues (0)

Important Issues (0)

Suggestions (0)

Strengths

• The workflow validates ramp ordering, positivity, and the supported maximum before making network measurements.
• Host source-port, container file-descriptor, and readable conntrack ceilings are enforced or explicitly downgraded to observations.
• The N=1 positive control and receiver-side per-tunnel survival classifier prevent a silent tuple or unmaintained tunnel from being reported as a clean capacity step.
• The Rust tests cover healthy survival, each deliberate failure mode, keep-alive minimums, and propagation of the measurement caveat.

Recommended Action

1. No Critical or Important issues remain from this pass.
2. Keep the ramp receipt and candidate _astats qualification evidence attached to the workflow run before using higher withheld steps.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: 627da00

Critical Issues (0)

Important Issues (1)

• [pr-review-toolkit:errors] .github/workflows/amt-public-vantage-probe.yml:332-350 — The host-ceiling guard compares the requested N with the total conntrack table size and total ephemeral-port span, not the currently available capacity. Existing flows on the GitHub runner can consume either resource, so a ramp can pass this preflight and then report a shortfall caused by runner NAT/source-port exhaustion as if it were relay state. The receipt records nf_conntrack_max and the port range but does not measure occupancy or reserve an actually-free budget.
  • Measure available conntrack entries and usable source ports immediately before the ramp, or fail the run when current occupancy cannot be established; compare the largest requested N against that available budget and label the result non-attributable otherwise.

Suggestions (0)

Strengths

• The workflow validates non-empty, numeric, strictly ascending ramp steps and bounds them to the validated maximum.
• The SHA-image fallback now builds from the checked-out tree and verifies the embedded client revision before measurement.
• The N=1 positive control and receiver-side survival classifier prevent a silent tuple or unmaintained tunnel from being reported as clean capacity.
• The Rust tests cover healthy survival, deliberate failure modes, keep-alive minimums, and propagation of the measurement caveat.

Recommended Action

1. Fix the Important issue before relying on a ramp result as a relay capacity measurement.
2. Re-run the ramp only after the preflight can distinguish total host limits from available host capacity.

[allyblockcast]

@ally please re-review at head a660d4c — focus on the new verdict gate and on whether I have retired your Important correctly rather than dodged it.

Your Important (host-ceiling guard measures totals, not available capacity) is correct, and I have not implemented the occupancy measurement. I took the second branch you offered — "label the result non-attributable otherwise" — because while checking it I found a dominant defect that makes the occupancy fix insufficient on its own:

alive never witnessed the relay at all. It is state == Active && keepalives_sent >= 1; all three not-alive reasons are client-local, and keepalives_sent increments after this process's send_to returns. Meanwhile run_tunnels builds every gateway with builder(relay), so all N share the host's default source address — and linux-amt as deployed matches a tunnel on the outer source address alone (tunnel->addr.ip4 == iph->saddr, amt_request_handler), overwriting source_port from the newest Request. So N gateways from one address are one relay tunnel entry, while every aliased gateway still reads Active and still sends keep-alives (its Updates fail the relay's MAC check silently). alive reports N; the relay holds 1.

So a conntrack/source-port fix would have made a non-attributable number precisely non-attributable. Both your failure mode and mine land in the same place — the run cannot be sized against — and the gate now says so for the stronger reason, superseding every branch including clean-to-cap.

Worth your attention: the existing positive control cannot catch this. each_deliberate_failure_mode_reduces_the_alive_count fails a gateway by stopping it sending, which is the one thing the classifier can see. It tests the plumbing, not the confound.

Controls on the new gate: shared+clean-to-cap fires · field-absent (the real 21:12Z artifact shape, alive 248/256) fires · partial provisioning fires · distinct sources does NOT fire and falls through to the real verdict, so the gate is not permanently clamped. tunnels_mode asserts distinct_outer_sources == 1 exactly, so a future per-source rig must fail that test and set the field deliberately.

If you think the occupancy measurement should still land on top of this, say so and I will file it — but I would rather not spend it on a mode I am retiring for the state claim.