Skip to content

fix(opentelemetry): guard span:finish() against shutdown-time export errors - #13985

Open
akashchamp wants to merge 1 commit into
apache:masterfrom
akashchamp:fix-opentelemetry-shutdown-span-finish-crash
Open

akashchamp wants to merge 1 commit into
apache:masterfrom
akashchamp:fix-opentelemetry-shutdown-span-finish-crash

Conversation

@akashchamp

@akashchamp akashchamp commented Sep 23, 2026 •

Copy link
Copy Markdown

Description

The vendored opentelemetry-lua dependency (pinned opentelemetry-lua = 0.2-6) has a bug in batch_span_processor.lua's create_timer(): when ngx.timer.at() fails because the worker is exiting — which happens near the end of every graceful shutdown/deploy — it falls back to a synchronous span export (self:flush_all()), which opens a cosocket.

Since span:finish() (called from apisix/plugins/opentelemetry.lua's _M.log and create_child_span) runs inline inside body_filter_by_lua*/log_by_lua*, that synchronous cosocket call trips APISIX's own phase guard in apisix/patch.lua and crashes with an uncaught Lua error:

2026/09/20 09:31:05 [error] 70#70: *1277095 failed to run log_by_lua*: /usr/local/apisix/apisix/patch.lua:406: API disabled in the context of log_by_lua*

Because this only requires the worker to be exiting — not any specific plugin config — it fires on effectively every rolling deploy of any release with tracing enabled on any route.

The real fix belongs upstream in opentelemetry-lua (already filed/fixed there: yangxikun/opentelemetry-lua#107, fix at yangxikun/opentelemetry-lua#106). Until APISIX bumps the pinned version to pick that up, this PR adds the defense-in-depth guard suggested in the issue: wrap span:finish() in pcall so this class of "third-party tracer code doing something disallowed in a restricted phase" can't crash the request phase, even though it can't be fully prevented from the plugin side alone.

Which issue(s) this PR fixes:

Fixes #13980

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change — see "How this was tested" below for why I held off on a checked-in t/*.t test and what I verified instead
  • I have updated the documentation to reflect this change — this is an internal defense-in-depth guard with no user-visible behavior change, so I didn't see a doc to update; happy to add one if reviewers want it documented
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

How this was tested

  • make lint (luacheck over apisix/t/lib, LuaJIT compile-style check via lj-releng, test-file style check) passes clean: Total: 0 warnings / 0 errors in 408 files.

  • This environment doesn't have a running OpenResty/test-nginx harness available, so I could not run make test's t/plugin/opentelemetry*.t suite directly, and a genuine worker-shutdown race isn't something the existing Test::Nginx .t framework can trigger deterministically (there's no existing precedent for mocking ngx.timer.at()/ngx.worker.exiting() failures anywhere in t/). Instead, I reproduced the bug and verified the fix directly against the real vendored dependency: I fetched the actual, unmodified opentelemetry-lua v0.2.6 source (batch_span_processor.lua, global.lua, metrics_reporter.lua — the exact version pinned in apisix-master-0.rockspec), mocked only the ngx.* surface it touches (including an ngx.socket.tcp() that raises the exact apisix/patch.lua phase-guard error), and confirmed with a plain Lua script that:

    1. Before the fix (calling span:finish() directly), the phase-guard error escapes uncaught — reproducing the issue exactly, including the verbatim error string from the report.
    2. After the fix (finish_span(), the new pcall wrapper), the same error is caught and logged as a warning instead of propagating — no crash.
    3. Sanity check: on the normal (non-shutdown) path, where nothing fails, the guard is a no-op — the exporter still runs and no warning is logged.

    All three checks pass (ALL CHECKS PASSED). I'm glad to add this as a checked-in test if there's a preferred location/pattern for it — I didn't want to guess at one given there's no existing precedent in t/ for this kind of timer/shutdown mocking.

Limitations

…errors

The vendored opentelemetry-lua's batch_span_processor falls back to a
synchronous flush_all() when ngx.timer.at() fails to schedule its
background flush timer, which happens near the end of every graceful
worker shutdown. That synchronous flush opens a cosocket, which trips
APISIX's own phase guard (apisix/patch.lua) when span:finish() runs
inside a restricted phase such as log_by_lua* or body_filter_by_lua*,
crashing the last span(s) on effectively every rolling deploy that has
the opentelemetry plugin's tracing enabled.

Wrap span:finish() in pcall as defense in depth, since this class of
"third-party tracer code doing something disallowed in a restricted
phase" can't be fully prevented from the plugin side alone. The
underlying fix belongs upstream (yangxikun/opentelemetry-lua#106/apache#107);
this only stops it from crashing the request phase in the meantime.

Fixes apache#13980
@akashchamp
akashchamp force-pushed the fix-opentelemetry-shutdown-span-finish-crash branch from 393ac32 to f03d8c2 Compare September 25, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

opentelemetry plugin: uncaught Lua error on worker shutdown via vendored opentelemetry-lua batch_span_processor

1 participant