fix(opentelemetry): guard span:finish() against shutdown-time export errors - #13985
Open
akashchamp wants to merge 1 commit into
Open
akashchamp wants to merge 1 commit into
akashchamp wants to merge 1 commit into
Conversation
…errors The vendored opentelemetry-lua's batch_span_processor falls back to a synchronous flush_all() when ngx.timer.at() fails to schedule its background flush timer, which happens near the end of every graceful worker shutdown. That synchronous flush opens a cosocket, which trips APISIX's own phase guard (apisix/patch.lua) when span:finish() runs inside a restricted phase such as log_by_lua* or body_filter_by_lua*, crashing the last span(s) on effectively every rolling deploy that has the opentelemetry plugin's tracing enabled. Wrap span:finish() in pcall as defense in depth, since this class of "third-party tracer code doing something disallowed in a restricted phase" can't be fully prevented from the plugin side alone. The underlying fix belongs upstream (yangxikun/opentelemetry-lua#106/apache#107); this only stops it from crashing the request phase in the meantime. Fixes apache#13980
akashchamp
force-pushed
the
fix-opentelemetry-shutdown-span-finish-crash
branch
from
September 25, 2026 17:59
393ac32 to
f03d8c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The vendored
opentelemetry-luadependency (pinnedopentelemetry-lua = 0.2-6) has a bug inbatch_span_processor.lua'screate_timer(): whenngx.timer.at()fails because the worker is exiting — which happens near the end of every graceful shutdown/deploy — it falls back to a synchronous span export (self:flush_all()), which opens a cosocket.Since
span:finish()(called fromapisix/plugins/opentelemetry.lua's_M.logandcreate_child_span) runs inline insidebody_filter_by_lua*/log_by_lua*, that synchronous cosocket call trips APISIX's own phase guard inapisix/patch.luaand crashes with an uncaught Lua error:Because this only requires the worker to be exiting — not any specific plugin config — it fires on effectively every rolling deploy of any release with tracing enabled on any route.
The real fix belongs upstream in
opentelemetry-lua(already filed/fixed there: yangxikun/opentelemetry-lua#107, fix at yangxikun/opentelemetry-lua#106). Until APISIX bumps the pinned version to pick that up, this PR adds the defense-in-depth guard suggested in the issue: wrapspan:finish()inpcallso this class of "third-party tracer code doing something disallowed in a restricted phase" can't crash the request phase, even though it can't be fully prevented from the plugin side alone.Which issue(s) this PR fixes:
Fixes #13980
Checklist
t/*.ttest and what I verified insteadHow this was tested
make lint(luacheck overapisix/t/lib, LuaJIT compile-style check vialj-releng, test-file style check) passes clean:Total: 0 warnings / 0 errors in 408 files.This environment doesn't have a running OpenResty/
test-nginxharness available, so I could not runmake test'st/plugin/opentelemetry*.tsuite directly, and a genuine worker-shutdown race isn't something the existingTest::Nginx.tframework can trigger deterministically (there's no existing precedent for mockingngx.timer.at()/ngx.worker.exiting()failures anywhere int/). Instead, I reproduced the bug and verified the fix directly against the real vendored dependency: I fetched the actual, unmodifiedopentelemetry-luav0.2.6 source (batch_span_processor.lua,global.lua,metrics_reporter.lua— the exact version pinned inapisix-master-0.rockspec), mocked only thengx.*surface it touches (including anngx.socket.tcp()that raises the exactapisix/patch.luaphase-guard error), and confirmed with a plain Lua script that:span:finish()directly), the phase-guard error escapes uncaught — reproducing the issue exactly, including the verbatim error string from the report.finish_span(), the newpcallwrapper), the same error is caught and logged as a warning instead of propagating — no crash.All three checks pass (
ALL CHECKS PASSED). I'm glad to add this as a checked-in test if there's a preferred location/pattern for it — I didn't want to guess at one given there's no existing precedent int/for this kind of timer/shutdown mocking.Limitations
span:finish()call sites APISIX's ownopentelemetry.luaplugin controls. It does not fix the root cause in the vendored library, which still needs the pinnedopentelemetry-luaversion bumped once fix: don't force a synchronous export when create_timer fails on worker exit yangxikun/opentelemetry-lua#106 is released.