Skip to content

test(litellm): Require filelock>=4.0.3 on Python 3.10/3.11 - #8026

Merged
sentrivana merged 1 commit into
masterfrom
fix/litellm-py310-settrace-flake
Oct 14, 2026
Merged

sentrivana merged 1 commit into
masterfrom
fix/litellm-py310-settrace-flake

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Description

This fixes the AI (3.10) litellm flake. The test that fails changes from run to run. Recent example: test_binary_content_encoding_mixed_content[False-False], which failed 7 times on master between Sep 30 and Oct 13 (job). Other failures were in test_no_integration[False-False] and test_binary_content_encoding_image_url[True-False]. All 10 litellm failures I looked at on master have the same cause:

/usr/lib/python3.10/threading.py:1011: in _bootstrap_inner
    _sys.settrace(_trace_hook)
E   RuntimeError: Cannot install a trace function while another trace function is being installed

Root cause

  • On CPython < 3.12, _PyEval_SetTrace uses a static int reentrant flag shared by the whole process. The flag stays set while audit hooks run (python/cpython#98257, fixed in 3.12).
  • filelock 3.30.0 through 4.0.2 registers a Python audit hook (sys.addaudithook(_audit_fork_safety)) when it is imported. The py3.10 litellm envs pin filelock==3.32.4, and huggingface_hub imports it during the first test.
  • pytest-cov calls threading.settrace, so every new thread calls sys.settrace() while it starts up. If the GIL switches while another thread is inside filelock's hook, the new thread's settrace() raises and the thread dies before it runs.
  • The tests create a new single-worker executor (reset_litellm_executor), and LiteLLM finishes the gen_ai.chat span in a success callback submitted to it. A dead worker means the callback never runs, so the test gets len(chat_spans) == 0.

Fix
filelock 4.0.3 fixed this upstream: it no longer registers the hook on CPython < 3.12 (_SETTRACE_SURVIVES_AUDIT_HOOKS). This PR adds a py3.10,py3.11 rule, filelock>=4.0.3, to the litellm config, then regenerates tox.ini with scripts/generate-test-files.sh --only litellm.

  • Only the four py3.10-litellm-* blocks change; they now pin filelock==4.1.0. py3.12 and later are unaffected and unchanged.
  • Re-resolving those four blocks also moved other transitive pins, for example anthropic 1.0.0 → 1.13.0.
  • releases.jsonl is unchanged. --only would have pruned it, so I restored it. package_dependencies.jsonl only gains the 3 new resolution entries.

No retries, sleeps, skipped tests or weaker assertions.

Reproduction: getsentry/repro#60
It is a standalone script with no litellm and no pytest. On Python 3.10 with filelock 3.32.4, it lost 14 of 32,000 callbacks in one run, and 3 of 5 shorter runs (16,000 callbacks each) reproduced it. Under the same conditions it lost 0 of 32,000 with filelock 4.0.3, and 0 on Python 3.12.

Original suite, run locally (py3.10-litellm-v1.83.14, with coverage):

  • With default settings: 0 of 20 runs failed.
  • With GIL switching set to happen more often (sys.setswitchinterval(1e-6) via a -p plugin; no test code changed): 4 of 20 runs hit the RuntimeError, and 3 of 20 failed.

This is a draft until validation of the fix against both the repro and the original suite is posted here.

Possible follow-up, not changed here: py3.11-anthropic-v0.16.0 also pins filelock==3.32.4.

Issues

via Junior system actor event.

--

View Junior Session [Sentry]

filelock 3.30.0-4.0.2 registers a Python audit hook on import. On CPython
< 3.12, sys.settrace() is guarded by a process-wide reentrancy flag that
stays set while audit hooks run (python/cpython#98257). With coverage
installing a tracer for every thread, a freshly started ThreadPoolExecutor
worker can die in Thread._bootstrap with "Cannot install a trace function
while another trace function is being installed". LiteLLM finishes the
gen_ai.chat span in a thread pool success callback, so the span is lost
and the test sees zero chat spans.

filelock 4.0.3 skips the audit hook on affected interpreters. Pin it for
the py3.10/py3.11 litellm envs and regenerate tox.ini for litellm.

Refs #7014
@dingsdax
dingsdax marked this pull request as ready for review October 14, 2026 09:01
@dingsdax
dingsdax requested a review from a team as a code owner October 14, 2026 09:01
@github-actions

Copy link
Copy Markdown
Contributor

Coverage Results 📊

✅ 133666 passed | ⏭️ 7276 skipped | Total: 140942 | Pass Rate: 94.84% | Execution Time: 452m 39s

📊 Comparison with Base Branch

Metric Change
Total Tests —
Passed Tests 📈 +1
Failed Tests 📉 -1
Skipped Tests —

➖ Removed Tests (1)

View removed tests
  • test_segment_uses_downsampled_rate
    • File: tests.test_monitor

All tests are passing successfully.

✅ Patch coverage is 100.00% (no changed executable lines found; target 80%).
Project statement coverage is 90.23% (unchanged from base (2ffd383) to head (9663dec)).

Coverage diff
@@            Coverage Diff             @@
##        master     #8026       +/-##
==========================================
  Coverage    90.23%    90.23%        —%
==========================================
  Files          202       202         —
  Tracked lines     26659     26660        +1
  Branches      9926      9926         —
==========================================
+ Hits         24055     24056        +1
  Misses        2604      2604         —
  Partials      1510      1510         —

Generated by Coverage Action

@sentrivana
sentrivana merged commit c96bbef into master Oct 14, 2026
168 checks passed
@sentrivana
sentrivana deleted the fix/litellm-py310-settrace-flake branch October 14, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resolve RuntimeError: Cannot install a trace function while another trace function is being installed

1 participant