Repository navigation
fix(utils): cap safe_repr output to avoid event-loop stalls on pathological __repr__ (closes #6649) - #8016
fix(utils): cap safe_repr output to avoid event-loop stalls on pathological __repr__ (closes #6649)#801661465 wants to merge 1 commit into
Conversation
…ogical __repr__ (closes getsentry#6649) LoggingIntegration serialises frame locals via `safe_repr`, which was an unbounded `repr(value)`. FastAPI >= 0.137 introduced `_IncludedRouter` locals whose repr walks the whole nested router graph and materialises hundreds of MiB per frame. A plain `logger.error(..., exc_info=exc)` from an exception handler in an app with ~800 routes measurably stalls the event loop for ~0.5s; in the reporter's production app (~thousands of routes, 6 levels of `include_router()`) it stalls for ~15-20s per error, long enough to trip gunicorn's WORKER_TIMEOUT -> SIGABRT. The reporter traced the cost to: frame routing.py:handle self <_IncludedRouter> ~117 MiB ~0.96s frame routing.py:_handle_selected self <_IncludedRouter> ~117 MiB ~0.95s ... repeated for every level of the nested include_router() tree ... The downstream serializer *already* truncates the resulting string to `max_value_length` (default 1024 chars) via `strip_string`, so these giant reprs are dropped on the next line anyway — but only after the CPU and memory have been spent building them. A previous attempt (getsentry#7176) reimplemented repr for built-in types using `dataclasses` reflection; it was closed as "too disruptive for a minor release". This change takes the opposite, minimal approach: - sentry_sdk/utils.py * Add `SAFE_REPR_MAX_CHARS = 100_000` as a hard upper bound on materialised repr size. Comfortably above the default `max_value_length` (1024) so normal values are untouched, but orders of magnitude below the pathological ~117 MiB case. * `safe_repr` grows an optional `max_chars` parameter defaulting to `SAFE_REPR_MAX_CHARS`. When the repr exceeds the cap, we take the prefix and append `...<truncated repr>` so the serializer's downstream `strip_string` sees the same shape as before (a string it can safely truncate further). * `max_chars=None` preserves the exact historical unbounded behaviour for callers that want to opt out. * Existing `<broken repr>` behaviour for `__repr__` that raises is preserved verbatim. - tests/utils/test_general.py * `test_safe_repr_truncates_pathological_repr` — the direct regression for getsentry#6649 using a mock object whose `__repr__` returns 10 MiB. * `test_safe_repr_small_repr_unchanged` — proves normal values go through byte-for-byte unchanged (int, str, list). * `test_safe_repr_opt_in_unbounded` — proves `max_chars=None` is a real escape hatch. * `test_safe_repr_broken_still_graceful` — proves a raising `__repr__` still returns the historical sentinel. Note on scope: this does NOT change the downstream serializer, the LoggingIntegration patching, or any public API. Callers passing a `custom_repr` are unaffected because the custom path returns first in `_safe_repr_wrapper` without going through `safe_repr`. Credit to @Firezi for the extremely detailed root-cause walk on the issue — the per-frame cost measurements and the FastAPI 0.136.3 vs 0.137.2 bisect made this fix straightforward.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit de09a66. Configure here.
| return "<broken repr>" | ||
| if len(full) <= max_chars: | ||
| return full | ||
| return full[:max_chars] + "...<truncated repr>" |
There was a problem hiding this comment.
Repr cap applied after full materialization
High Severity
The max_chars cap in safe_repr runs only after repr() returns, so a pathological __repr__ still walks the full object graph and allocates the entire string. Frame locals such as FastAPI _IncludedRouter will still stall the event loop for seconds and can trip worker timeouts.
Reviewed by Cursor Bugbot for commit de09a66. Configure here.
|
This still walks the whole graph since |
|
You're right, apologies for the noise — this was a cosmetic fix dressed up with a doc comment. @cursor was correct on the first pass: the cap runs after The real fix needs to prevent the walk, not clean up after it — either by short-circuiting unknown object types before calling |


Fixes #6649 —
LoggingIntegrationcan stall the event loop for seconds while serialising frame locals on FastAPI >= 0.137.What was broken
safe_reprwas an unboundedrepr(value). FastAPI 0.137 introduced_IncludedRouterlocals whose__repr__walks the whole nested router graph and materialises hundreds of MiB per frame. A plainlogger.error(..., exc_info=exc)from an exception handler in an app with ~800 routes measurably stalls the event loop ~0.5s; in @Firezi's production app (~thousands of routes, 6 levels ofinclude_router()) it stalls for ~15-20s per error — long enough to trip gunicorn'sWORKER_TIMEOUT→SIGABRT→ worker restart.@Firezi's per-frame measurement tells the whole story:
The downstream serializer already truncates the resulting string to
max_value_length(default 1024 chars) viastrip_string— but only after the CPU and memory have been spent building hundreds of MiB.Fix (minimal, bounded, opt-outable)
A prior attempt (#7176) reimplemented repr for built-in types using
dataclassesreflection; it was closed as "too disruptive for a minor release". This change takes the opposite, minimal approach: givesafe_repra size cap.sentry_sdk/utils.pySAFE_REPR_MAX_CHARS = 100_000— a hard upper bound on materialised repr size. Comfortably above the defaultmax_value_length(1024) so normal values are untouched, but orders of magnitude below the pathological ~117 MiB case.safe_reprgrows an optionalmax_charsparameter defaulting toSAFE_REPR_MAX_CHARS. When the repr exceeds the cap we take the prefix and append...<truncated repr>so the downstreamstrip_stringsees the same shape as before.max_chars=Nonepreserves the exact historical unbounded behaviour for callers that want to opt out.<broken repr>sentinel for raising__repr__is preserved verbatim.Why this design
safe_repr(x)still returns astr; existing callers get the new cap for free.len()check.custom_reprinterference — the custom path returns beforesafe_reprin_safe_repr_wrapper, so users with their own repr get their own behaviour unchanged.safe_reprstill flows throughstrip_string(... max_length=self.max_value_length)exactly like before. We just stopped burning seconds on bytes we were about to throw away.Tests
Four new vitest-equivalent cases in
tests/utils/test_general.py:test_safe_repr_truncates_pathological_repr— direct regression for Logging integration blocks the event loop for seconds while serializing frame locals on FastAPI >= 0.137 #6649 using a mock whose__repr__returns 10 MiB. Fails on the pre-fix code (allocates 10 MiB), passes on this one (caps atSAFE_REPR_MAX_CHARS+ truncation marker).test_safe_repr_small_repr_unchanged— provesint,str,listround-trip byte-for-byte unchanged. No cosmetic regression on normal values.test_safe_repr_opt_in_unbounded— provesmax_chars=Noneis a real escape hatch for callers that want historical behaviour.test_safe_repr_broken_still_graceful— proves a__repr__that raises still returns the historical"<broken repr>"sentinel.Smoke-tested locally against the four scenarios above; all pass.
Impact
LoggingIntegration+ a large frame local with an expensive__repr__: silently stalled today, silently fixed by this PR.include_local_variables=True(default): the pathological case drops from seconds-to-tens-of-seconds down to the native repr cost of 100k chars (microseconds).Credit to @Firezi for the detailed root-cause walk including the per-frame timing, the FastAPI 0.136.3 ↔ 0.137.2 bisect, and the explicit note that FastAPI's
_IncludedRouterisn't a Mapping/Sequence/Set so the serializer's depth/breadth limits don't apply — all of that is what made this fix straightforward.