cg_origin_egress exceeds the 1M-instruction verifier budget on kernel 6.6 (Blacksmith): container egress and TLS SNI enforcement silently off

[matthewdevenny]

Symptom

On Blacksmith runners (Firecracker VM, kernel 6.6.141, BTF present, TCX attach works) the cgroup egress program does not load, so --container-egress enforce and --tls-sni enforce-pinned are both silently off for the job. From a v2.0.0-rc.9 run on blacksmith-2vcpu-ubuntu-2404 (madhokie-io/dockcmd, run 35283540609):

::warning::Container egress hook disabled — enforcement falls back to TC egress only target_mode=enforce error=failed to load origin BPF objects: assign values: field CgOriginEgress: program cg_origin_egress: load program: argument list too long: BPF program is too large. Processed 1000001 insn (1315 line(s) omitted)
::warning::--tls-sni requested but the cgroup egress hook is not running — L7 SNI enforcement is OFF and shared-edge IPs stay L4-only

Everything else in the run is healthy: step attribution works, TC enforcement works, DNS gating works. The job passes its CargoWall check with the operator believing container egress and SNI pinning are enforced.

Cause

cg_origin_egress sits at ~705k processed instructions on the 6.17 Azure kernel CI verifies against (see the notes in [[design.md]] / the L7 SNI work). The 6.6 verifier explores the same program past the 1,000,000-instruction limit — older verifiers prune fewer states in the SNI/QUIC byte loops. The Lima gate is pinned to 6.17 precisely because 6.8 was more forgiving than CI; 6.6 turns out to be less forgiving, and nothing in CI measures it.

6.6 is a current LTS and the kernel on Blacksmith today; GitLab SaaS shared runners are on 5.15, which will be worse.

Ask

1. Add a verifier-budget gate for the oldest kernels we claim to support (at minimum 6.6, ideally 5.15 for the GitLab preset) — a matrix job with a 6.6 kernel, or a vmtest-style run — and record the processed-instruction count per program so regressions are visible before release.
2. Bring cg_origin_egress under budget on 6.6: split the SNI/QUIC parsing into a tail call or a separate program, bound the byte loops more tightly, or gate the L7 parse behind a rodata flag so an L4-only build of the program loads where the L7 one cannot (the L4 container-egress enforcement would then survive on 6.6 even if SNI does not).
3. Make the degradation louder than a ::warning::. When the operator asked for container-egress: enforce or tls-sni: enforce-pinned and the hook cannot load, the job summary and the dashboard should say those postures were not applied; today it is only visible in the step log.

Related

• Kernel floor and clsact fallback: Add legacy TC (clsact) fallback so cargowall works on kernel 5.8+ #58
• L7 SNI enforcement: Add L7 sni verification #118; byte-loop verifier cost notes in the repo memory/design docs

[matthewdevenny]

Workaround options, in the order I'd try them

The limit is on verifier states, not program size; the L7 walk (coalesced QUIC loop, unrolled varint, 64-segment GSO tail scan, TLS extension walk) is all __always_inline into every path of cg_origin_egress, and the 6.6 verifier prunes far fewer of those states than 6.7–6.9 do.

1. Measure on 6.6 first. A CI job on blacksmith-2vcpu-ubuntu-2404 loading the programs with ebpf.LogLevelStats gives per-program processed N insns on a real 6.6 kernel and is the oldest-kernel gate this issue asks for. Nothing below should be sized without that number.
2. Rodata L7 gate + automatic L4-only reload. const volatile flag for the L7 path; on a too-large load error, reload once with it cleared and warn. Restores --container-egress enforce on 6.6 immediately; SNI stays off there until 3/4 land.
3. Global functions. Make the QUIC packet parser, the TLS walk and l7_adjudicate non-static __noinline functions taking the skb ctx and sized stack structs. Verified once in isolation (kernel 5.10+), so callers stop multiplying the state count. Biggest single lever; portable to the 5.15 GitLab floor.
4. bpf_loop for the bounded scans (coalesced walk, varint, GSO tail). Callback verified once instead of unrolled; 5.17+, so feature-detect and keep behind the gate from 2.
5. Tail-call split. L7 adjudication as its own program reached by bpf_tail_call after the L4 verdict: separate 1M budget, and a failed L7 load leaves L4 intact by construction (empty prog-array slot falls through).
6. State-space trims. barrier_var on offsets after bounds checks; revisit L7_GSO_MAX_SEGS=64 and L7_QUIC_MAX_COALESCED=4 — the design already punts undecidable flights to userspace, so a dumber kernel parser on old kernels is a legitimate outcome.

[matthewdevenny]

Scoping note: 2.0.0 is still in release candidates, so no stopgap is planned. Step 2 above (rodata gate + L4-only reload) is not a separate deliverable; the structural fix — global functions and bpf_loop for the scans, measured on a 6.6 gate, with the tail-call split held in reserve — is the target for GA. The property that L4 container-egress loads even where L7 cannot should still fall out of the design (the tail-call split gives it by construction), together with louder reporting when a requested posture is not applied.

[matthewdevenny]

Measured with the tool from #139 (scripts/ci/verifierbudget), same commit on both kernels. Blacksmith run: madhokie-io/dockcmd PR #3, run 35285488342 on blacksmith-2vcpu-ubuntu-2404.

Program	6.17.0-1022-azure (CI / Lima)	6.6.141 (Blacksmith)
cg_origin_egress	657,658 (65.8%)	1,000,001 — rejected
tc_egress	5,546	5,503
step_fork	1,220	1,240
step_task_iter	1,095	1,095
cgroup connect/sendmsg, cg_sock_create, step_exit, tc_ingress	≤ 60	≤ 60

So the two verifiers agree on every program but one, and on that one 6.6 explores at least 1.52× the states 6.17 does before giving up — the true 6.6 cost is unknown because the verifier stops at the limit. Every other program has three orders of magnitude of headroom; the whole problem is the L7 walk inside cg_origin_egress, which is where the global-function / bpf_loop work should go. The dockcmd workflow can be dispatched with any cargowall_ref to measure a branch on 6.6 before it lands.