Repository navigation
test(litellm): Require filelock>=4.0.3 on Python 3.10/3.11 - #8026
Merged
Merged
Conversation
filelock 3.30.0-4.0.2 registers a Python audit hook on import. On CPython < 3.12, sys.settrace() is guarded by a process-wide reentrancy flag that stays set while audit hooks run (python/cpython#98257). With coverage installing a tracer for every thread, a freshly started ThreadPoolExecutor worker can die in Thread._bootstrap with "Cannot install a trace function while another trace function is being installed". LiteLLM finishes the gen_ai.chat span in a thread pool success callback, so the span is lost and the test sees zero chat spans. filelock 4.0.3 skips the audit hook on affected interpreters. Pin it for the py3.10/py3.11 litellm envs and regenerate tox.ini for litellm. Refs #7014
dingsdax
marked this pull request as ready for review
October 14, 2026 09:01
Contributor
Coverage Results 📊✅ 133666 passed | ⏭️ 7276 skipped | Total: 140942 | Pass Rate: 94.84% | Execution Time: 452m 39s 📊 Comparison with Base Branch
➖ Removed Tests (1)View removed tests
All tests are passing successfully. ✅ Patch coverage is 100.00% (no changed executable lines found; target 80%). Coverage diff@@ Coverage Diff @@
## master #8026 +/-##
==========================================
Coverage 90.23% 90.23% —%
==========================================
Files 202 202 —
Tracked lines 26659 26660 +1
Branches 9926 9926 —
==========================================
+ Hits 24055 24056 +1
Misses 2604 2604 —
Partials 1510 1510 —Generated by Coverage Action |
sentrivana
approved these changes
Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This fixes the
AI (3.10)litellm flake. The test that fails changes from run to run. Recent example:test_binary_content_encoding_mixed_content[False-False], which failed 7 times on master between Sep 30 and Oct 13 (job). Other failures were intest_no_integration[False-False]andtest_binary_content_encoding_image_url[True-False]. All 10 litellm failures I looked at on master have the same cause:Root cause
_PyEval_SetTraceuses astatic int reentrantflag shared by the whole process. The flag stays set while audit hooks run (python/cpython#98257, fixed in 3.12).filelock3.30.0 through 4.0.2 registers a Python audit hook (sys.addaudithook(_audit_fork_safety)) when it is imported. The py3.10 litellm envs pinfilelock==3.32.4, andhuggingface_hubimports it during the first test.threading.settrace, so every new thread callssys.settrace()while it starts up. If the GIL switches while another thread is inside filelock's hook, the new thread'ssettrace()raises and the thread dies before it runs.reset_litellm_executor), and LiteLLM finishes thegen_ai.chatspan in a success callback submitted to it. A dead worker means the callback never runs, so the test getslen(chat_spans) == 0.Fix
filelock4.0.3 fixed this upstream: it no longer registers the hook on CPython < 3.12 (_SETTRACE_SURVIVES_AUDIT_HOOKS). This PR adds apy3.10,py3.11rule,filelock>=4.0.3, to the litellm config, then regeneratestox.iniwithscripts/generate-test-files.sh --only litellm.py3.10-litellm-*blocks change; they now pinfilelock==4.1.0. py3.12 and later are unaffected and unchanged.anthropic1.0.0 → 1.13.0.releases.jsonlis unchanged.--onlywould have pruned it, so I restored it.package_dependencies.jsonlonly gains the 3 new resolution entries.No retries, sleeps, skipped tests or weaker assertions.
Reproduction: getsentry/repro#60
It is a standalone script with no litellm and no pytest. On Python 3.10 with filelock 3.32.4, it lost 14 of 32,000 callbacks in one run, and 3 of 5 shorter runs (16,000 callbacks each) reproduced it. Under the same conditions it lost 0 of 32,000 with filelock 4.0.3, and 0 on Python 3.12.
Original suite, run locally (
py3.10-litellm-v1.83.14, with coverage):sys.setswitchinterval(1e-6)via a-pplugin; no test code changed): 4 of 20 runs hit theRuntimeError, and 3 of 20 failed.This is a draft until validation of the fix against both the repro and the original suite is posted here.
Possible follow-up, not changed here:
py3.11-anthropic-v0.16.0also pinsfilelock==3.32.4.Issues
RuntimeError: Cannot install a trace function while another trace function is being installed#7014via Junior system actor
event.--
View Junior Session [Sentry]