Evict idle pooled HTTP/2 connections; cover them in close_idle_connections! (#1331)

[quinnj]

Fixes #1331.

Root cause

Pooled HTTP/2 connections (client.h2_conns) had no idle eviction and no checkout validation — unlike the HTTP/1 Transport pool, which evicts anything idle past idle_timeout_ns (90s default). The checkout scan consulted only reactive flags (closed, conn_error, GOAWAY state); a NAT/load-balancer that silently drops an idle flow produces no event, so a 12-minutes-idle connection still looked perfectly available. The next request's HEADERS/DATA were written into the dead path, the kernel retransmitted until tcp_retries2 (~15.9 min on Linux), then killed the socket with ETIMEDOUT — delivered to the parked reader, hence HTTP/2 read loop failed … SystemError(read, 110). Every timing in the report matches. (1.x is immune because it has no HTTP/2; 2.x's h1 pool was equally immune — the gap was h2-only.)

A second bug compounded it: close_idle_connections! never touched the h2 pool — both the Client and no-arg forms delegated to the transport only, so the documented remedy (and the reporter's workaround) didn't reach the pool with the problem; the green workaround run passed for other reasons (a GOAWAY/RST arriving in time self-invalidates the connection — the pathology needs the drop to be silent).

Both reproduced deterministically on master with a frame server that answers once and then goes mute (stands in for the dead path): the pooled conn is handed out again, request 2 blocks indefinitely, and close_idle_connections! leaves the conn pooled.

Fix (design)

Idleness is a property of the connection's stream lifecycle, tracked where that lifecycle already changes, and evicted lazily at checkout (the h1 pool's established pattern — no background reaper, no new locks, no new API):

• H2Connection.idle_since_ns (guarded by the existing state_lock), stamped at creation and on each transition into the fully-idle state — last stream unregistered, last pending reservation released. A connection continuously serving traffic is never considered idle.
• _acquire_h2_conn! evicts connections idle past the transport's existing idle_timeout_ns — one knob, one semantic, same 90s default (comfortably inside any NAT window), 0 disables, exactly like h1.
• Culled connections are closed after h2_lock is released: close(H2Connection) waits for its read loop to exit, and that must not run under the pool lock. This also moves the pre-existing inline close of non-reusable connections out from under the lock — a latent acquire-path latency hazard.
• close_idle_connections!(client) (and the no-arg form) now also close pooled h2 connections with no in-flight streams and no pending reservations, keeping active ones — the documented remedy now works.

Deliberately not included: per-checkout h2 PING health checks (adds an RTT to every reuse and needs its own timeout machinery; Go ships this as an opt-in ReadIdleTimeout, off by default, for good reason). Eviction removes the hang class; keepalive PINGs can be a separate opt-in follow-up if wanted.

Behavior after the fix

The original repro now completes the post-idle request in ~20ms on a fresh connection instead of hanging ~16 minutes:

request 1: 200 in 0.015s
request 2: 200 in 0.019s   (fresh conn after eviction — NO HANG)
h2 pool: 1 -> 0 after close_idle_connections!

Tests

Three new testsets: end-to-end eviction against a mute-after-first-response server (second request must complete fast on a fresh connection, evicted conn closed); fresh-connection reuse + idle_timeout_ns=0 disables eviction (guards against over-eviction); close_idle_connections! closes idle h2 conns while keeping ones with pending reservations. Full h2 suite 181/181; transport 106, refused-stream retry 22, client 586 all green locally.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.14815% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 88.23%. Comparing base (7bb9b2c) to head (18244e5).

Files with missing lines	Patch %	Lines
src/http_client.jl	97.22%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1332      +/-   ##
==========================================
+ Coverage   88.07%   88.23%   +0.16%     
==========================================
  Files          30       30              
  Lines       11871    11921      +50     
==========================================
+ Hits        10455    10519      +64     
+ Misses       1416     1402      -14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.