Raise MemoryHigh to 150M to fit chardet's lazily-loaded tables

[mnot]

Follow-up to #425, which fixed two watchdog-stall causes but set MemoryHigh from a working-set figure that turned out to be incomplete. Production kept tripping the watchdog afterwards, most recently on Jul 29 with --debug off.

What was actually consuming the memory

Measured, not inferred. One 14 KB text/html; charset=utf-8 fetch through redbot's own HttpResource:

redbot loaded, before any fetch      48.0 MB
after fetching 14KB text/html        94.1 MB

+46 MB, permanent, from a single small text response. In isolation it's chardet's first detect() call: 19.5 → 65.6 MB, matching exactly. Subsequent detects add nothing.

The path is httplint's verify_charset, which calls chardet.detect(sample) for any text/* with a charset parameter (or +json). chardet then lazily loads 352 bigram models of 64 KB each — 22.0 MB of live tables, plus ~24 MB of decompression the allocator never returns. That 22.0 MB matches the chardet/__init__.py:83 line that had been sitting at the top of every production memory dump.

So the real floor is ~57M base + 46M chardet ≈ 103M, against a MemoryHigh of 96M. The kernel throttles in reclaim, the loop parks for tens of seconds, the watchdog fires. Production peaks clustered at 104–117M across every crash — with and without --debug — which is what that arithmetic predicts.

Concurrency is not a factor

Measured with chardet pre-warmed, so only per-request cost shows:

concurrent	small (14 KB)	large (5 MB)
1	+0.0 MB	+0.0 MB
5	+0.0 MB	+0.0 MB
20	+2.0 MB	+2.6 MB
50	+6.8 MB	—

~0.14 MB per in-flight request, near enough independent of body size. Twenty concurrent 5 MB fetches — 100 MB streamed — cost 2.6 MB total. Bodies genuinely stream: feed_content hashes and discards, redbot's sampler caps at 8 KB, httplint's at 8192 bytes.

The change

MemoryHigh 96M → 150M (~47M of headroom over the ~103M floor, enough for several hundred concurrent requests), MemoryMax 128M → 192M so a genuine runaway is still caught. The comment now records the measured breakdown and where each number came from.

It also corrects a claim I put in that comment in #425 — that --debug costs ~45M of tracemalloc overhead. That figure was never measured; it was chardet's 46M misattributed. The Jul 29 crash had --debug off and still peaked at 107.8M.

For reviewers

• Deployment-only change. No Python touched, so make test passing says nothing about it. The unit file is still not syntax-checked — systemd-analyze verify isn't available on the dev machine.
• Numbers are from macOS, where the base process measures ~48M against production's ~57M. The +46M chardet delta and the +0.14M/request slope should carry over, since both are allocation behaviour rather than platform overhead — but the absolute floor on Linux is the production figure, not 94M.
• 150M is headroom, not a measured ceiling. After deploy, memory.events should stop incrementing high; if it doesn't, the floor is higher than I've measured and this needs revisiting.
• Not attempted: reducing the 46M. include_encodings=[...] restricted to three encodings gave no reduction at all — chardet's @functools.cached loader pulls in every model before filtering. There's no lazy-per-encoding path in 7.4.3. One untested idea: ~24M of the 46M is freed-but-unreturned decompression memory, so malloc_trim(0) via ctypes might reclaim it on glibc. I couldn't test that on macOS and am not proposing it here.

Related

Filed mnot/httplint#155 for a separate bug found while measuring: verify_charset reports a false CHARSET_UNDECODABLE when a multi-byte character straddles the 8192-byte content_sample boundary, because the sample is truncated at a fixed byte count and then decoded with errors="strict".

---

🤖 Written by Claude Opus 4.8 in Claude Code. @mnot directed the investigation throughout, supplied every production journal excerpt and cgroup reading, proposed the 150M figure, and asked for the concurrency evaluation that informs it. Several earlier hypotheses in this investigation — a large-body memory spike, chardet being CPU-bound, tracemalloc being the dominant footprint — were wrong and were discarded once measured; the numbers above come from instrumented runs whose output is quoted verbatim. Not yet verified against the production host.