signal 11 (core dumped) 3 mins after making a call

[Shaun-3adesign]

Bug Overview

when we make call to a flask app we get a core dump around 3 mins after, this is present in versions 1.32.2, 1.33.0, 1.34.0, 1.34.2
the call we are making makes a insert or a delete to a mysql database using mysql-python-connector

Expected Behavior

should not core dump

Steps to Reproduce the Bug

run application and make a post or delete call

Environment Details

• Target deployment platform: local server in a docker container
• Target OS: Ubuntu Desktop 22.04
• Version of this project or specific commit: 1.34.2
• Version of any relevant project languages: Python 3.11.2, Docker version 24.0.7, build afdd53b

listener.json
config.json
Dockerfile.txt
docker-entrypoint.txt

Additional Context

debug log
server.txt

[ac000]

HI, thanks for the report.

when we make call to a flask app we get a core dump around 3 mins after,
this is present im versions 1.32.2, 1.33.0, 1.34.0, 1.34.2

Interesting, so IIUC, you can make a single request to the application, wait ~3mins and the the application process will crash?

Is this 100% the case and is it always after around 3 minutes?.

The good news is your getting a core dump (as I don't expect you have a handy reproducer).

Worked with coredumps before? I would really love a backtrace...

I'm going to assume you know where the coredumps are going, so next time you get one can you simply do

$ gdb /path/to/unitd /path/to/coredump
...
(gdb) bt

And paste the output. You may or may not get symbols displayed and you may need to install debuginfo packges for unit/python, but lets see what we get first.

[ac000]

Another quick check would be to see if the problem happens without threads, i.e. comment out "threads": 4, (or change it to 1) in the config.

[Shaun-3adesign]

HI, thanks for the report.

when we make call to a flask app we get a core dump around 3 mins after,
this is present im versions 1.32.2, 1.33.0, 1.34.0, 1.34.2

Interesting, so IIUC, you can make a single request to the application, wait ~3mins and the the application process will crash?

Is this 100% the case and is it always after around 3 minutes?.

The good news is your getting a core dump (as I don't expect you have a handy reproducer).

Worked with coredumps before? I would really love a backtrace...

I'm going to assume you know where the coredumps are going, so next time you get one can you simply do

$ gdb /path/to/unitd /path/to/coredump
...
(gdb) bt
And paste the output. You may or may not get symbols displayed and you may need to install debuginfo packges for unit/python, but lets see what we get first.

we can reproduce the issue very easily. thou the core dump files don't seem to be getting saved anywhere.

[Shaun-3adesign]

let me know if you need more as i've not done this before

(gdb) bt
#0 0x00006356a3f4a093 in ?? ()
#1 0x00006356a3f4a1c9 in ?? ()
#2 0x00006356a3f3200e in nxt_event_engine_start ()
#3 0x00006356a3f2fcc6 in ?? ()
#4 0x00007997955671f5 in start_thread (arg=) at ./nptl/pthread_create.c:442
#5 0x00007997955e6b00 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

also we tested the with 1 thread and the issue is still present

[Shaun-3adesign]

the flask server we are running is running an opanapi server just connecting to the swagger page causes the core dump. so not linked to doing a post.

2025/05/02 13:27:35 [info] 198#198 router started
2025/05/02 13:27:35 [info] 198#198 OpenSSL 3.0.15 3 Sep 2024, 300000f0
2025/05/02 13:27:35 [info] 199#199 "flask" prototype started
2025/05/02 13:27:35 [info] 200#200 "flask" application started
2025/05/02 13:27:36 [info] 226#226 "flask" application started
2025/05/02 13:27:36 [info] 252#252 "flask" application started
2025/05/02 13:27:36 [info] 278#278 "flask" application started
172.18.0.1 - - [02/May/2025:13:29:37 +0000] "GET /resource_manager/ui/ HTTP/1.1" 200 1498 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0" "0.032"
172.18.0.1 - - [02/May/2025:13:29:37 +0000] "GET /resource_manager/ui/swagger-ui.css HTTP/1.1" 200 143669 "https://localhost:20018/resource_manager/ui/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0" "0.012"
172.18.0.1 - - [02/May/2025:13:29:37 +0000] "GET /resource_manager/ui/swagger-ui-bundle.js HTTP/1.1" 200 1091405 "https://localhost:20018/resource_manager/ui/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0" "0.036"
172.18.0.1 - - [02/May/2025:13:29:38 +0000] "GET /resource_manager/ui/swagger-ui-standalone-preset.js HTTP/1.1" 200 337216 "https://localhost:20018/resource_manager/ui/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0" "0.004"
172.18.0.1 - - [02/May/2025:13:29:38 +0000] "GET /resource_manager/ui/favicon-32x32.png HTTP/1.1" 200 628 "https://localhost:20018/resource_manager/ui/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0" "0.004"
172.18.0.1 - - [02/May/2025:13:29:38 +0000] "GET /resource_manager/openapi.json HTTP/1.1" 200 55553 "https://localhost:20018/resource_manager/ui/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0" "0.063"
2025/05/02 13:32:34 [alert] 7#7 process 198 exited on signal 11 (core dumped)
2025/05/02 13:32:34 [info] 336#336 router started
2025/05/02 13:32:34 [info] 336#336 OpenSSL 3.0.15 3 Sep 2024, 300000f0
2025/05/02 13:32:34 [info] 345#345 "flask" prototype started
2025/05/02 13:32:34 [info] 346#346 "flask" application started
2025/05/02 13:32:34 [notice] 199#199 app process 252 exited with code 0
2025/05/02 13:32:34 [notice] 199#199 app process 278 exited with code 0
2025/05/02 13:32:34 [notice] 199#199 app process 200 exited with code 0
2025/05/02 13:32:34 [notice] 199#199 app process 226 exited with code 0
2025/05/02 13:32:34 [notice] 7#7 process 199 exited with code 0
2025/05/02 13:32:34 [info] 372#372 "flask" application started
2025/05/02 13:32:34 [info] 398#398 "flask" application started
2025/05/02 13:32:35 [info] 424#424 "flask" application started

[ac000]

Thanks for the backtrace and the log output.

It actually looks like it's the router process that's crashing and not the application process.

Unfortunately as feared you are missing most of the debug symbols. Lets see if we can fix that

Assuming you installed from packages, you should be able to get debuginfo by installing the unit-dbg & unit-python-3.11-dbg (just in case) packages.

Then (and you can probably use the same coredump) could you provide the output from both the bt & bt full gdb commands? Thanks.

[Shaun-3adesign]

i'n getting the following when install unit-python-3.11-dbg

root@ee67afd20b2b:/tmp# apt install unit-python-3.11-dbg
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package unit-python-3.11-dbg
E: Couldn't find any package by glob 'unit-python-3.11-dbg'

[Shaun-3adesign]

here is the backtrace from just installing unit-dbg

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `unit: router                                               '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  nxt_h1p_complete_buffers (task=task@entry=0x72a9e4001620, h1p=h1p@entry=0x6255219bc340, all=all@entry=1) at src/nxt_h1proto.c:1522
1522    src/nxt_h1proto.c: No such file or directory.
[Current thread is 1 (Thread 0x72a9ea5266c0 (LWP 305))]
(gdb) bt
#0  nxt_h1p_complete_buffers (task=task@entry=0x72a9e4001620, h1p=h1p@entry=0x6255219bc340, all=all@entry=1) at src/nxt_h1proto.c:1522
#1  0x000062550e4cb46f in nxt_h1p_shutdown (task=0x72a9e4001620, c=0x72a9e4001550) at src/nxt_h1proto.c:2129
#2  0x000062550e4b75c2 in nxt_event_engine_start (engine=0x6255219bc0b0) at src/nxt_event_engine.c:542
#3  0x000062550e4b5bf1 in nxt_thread_trampoline (data=0x6255219339e0) at src/nxt_thread.c:126
#4  0x000072a9ebb621f5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x000072a9ebbe1b00 in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt full
#0  nxt_h1p_complete_buffers (task=task@entry=0x72a9e4001620, h1p=h1p@entry=0x6255219bc340, all=all@entry=1) at src/nxt_h1proto.c:1522
        size = <optimized out>
        b = 0x0
        in = <optimized out>
        next = <optimized out>
        c = 0x0
#1  0x000062550e4cb46f in nxt_h1p_shutdown (task=0x72a9e4001620, c=0x72a9e4001550) at src/nxt_h1proto.c:2129
        timer = <optimized out>
        h1p = 0x6255219bc340
#2  0x000062550e4b75c2 in nxt_event_engine_start (engine=0x6255219bc0b0) at src/nxt_event_engine.c:542
        obj = 0x72a9e40015b8
        data = 0x0
        task = 0x72a9e4001620
        timeout = <optimized out>
        now = <optimized out>
        thr = <optimized out>
        handler = 0x62550e4b7a80 <nxt_timer_handler>
#3  0x000062550e4b5bf1 in nxt_thread_trampoline (data=0x6255219339e0) at src/nxt_thread.c:126
        __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {126074106308288, 255495596939513932, -1368, 11, 140734449295984, 126074097917952, -1812585321242455988, -4097801277494816692}, __mask_was_saved = 0}}, __pad = {0x72a9ea525a30, 0x0, 0x0, 0x0}}
        __cancel_routine = 0x62550e4b5a60 <nxt_thread_time_cleanup>
        __cancel_arg = <optimized out>
        __not_first_call = <optimized out>
        thr = <optimized out>
        link = 0x6255219339e0
        start = 0x62550e4c0c90 <nxt_router_thread_start>
#4  0x000072a9ebb621f5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x000072a9ebbe1b00 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

[Shaun-3adesign]

i'n getting the following when install unit-python-3.11-dbg

root@ee67afd20b2b:/tmp# apt install unit-python-3.11-dbg Reading package lists... Done Building dependency tree... Done Reading state information... Done E: Unable to locate package unit-python-3.11-dbg E: Couldn't find any package by glob 'unit-python-3.11-dbg'

fixed this the issue was the name its python3.11-dbg

[Shaun-3adesign]

bug trace with both unit-dbg & unit-python3.11-dbg installed

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `unit: router                                               '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  nxt_h1p_complete_buffers (task=task@entry=0x7a1904001620, h1p=h1p@entry=0x636b874f99f0, all=all@entry=1) at src/nxt_h1proto.c:1522
1522    src/nxt_h1proto.c: No such file or directory.
[Current thread is 1 (Thread 0x7a1913fff6c0 (LWP 306))]
(gdb) bt
#0  nxt_h1p_complete_buffers (task=task@entry=0x7a1904001620, h1p=h1p@entry=0x636b874f99f0, all=all@entry=1) at src/nxt_h1proto.c:1522
#1  0x0000636b7063246f in nxt_h1p_shutdown (task=0x7a1904001620, c=0x7a1904001550) at src/nxt_h1proto.c:2129
#2  0x0000636b7061e5c2 in nxt_event_engine_start (engine=0x636b874f9760) at src/nxt_event_engine.c:542
#3  0x0000636b7061cbf1 in nxt_thread_trampoline (data=0x636b87467350) at src/nxt_thread.c:126
#4  0x00007a191c65c1f5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007a191c6dbb00 in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt full
#0  nxt_h1p_complete_buffers (task=task@entry=0x7a1904001620, h1p=h1p@entry=0x636b874f99f0, all=all@entry=1) at src/nxt_h1proto.c:1522
        size = <optimized out>
        b = 0x0
        in = <optimized out>
        next = <optimized out>
        c = 0x0
#1  0x0000636b7063246f in nxt_h1p_shutdown (task=0x7a1904001620, c=0x7a1904001550) at src/nxt_h1proto.c:2129
        timer = <optimized out>
        h1p = 0x636b874f99f0
#2  0x0000636b7061e5c2 in nxt_event_engine_start (engine=0x636b874f9760) at src/nxt_event_engine.c:542
        obj = 0x7a19040015b8
        data = 0x0
        task = 0x7a1904001620
        timeout = <optimized out>
        now = <optimized out>
        thr = <optimized out>
        handler = 0x636b7061ea80 <nxt_timer_handler>
#3  0x0000636b7061cbf1 in nxt_thread_trampoline (data=0x636b87467350) at src/nxt_thread.c:126
        __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {134248128313024, 5039262364208667989, -1368, 11, 140726245626752, 134248119922688, -5630298242980307627, -8990892774217671339}, __mask_was_saved = 0}}, __pad = {0x7a1913ffea30, 0x0, 0x0, 0x0}}
        __cancel_routine = 0x636b7061ca60 <nxt_thread_time_cleanup>
        __cancel_arg = <optimized out>
        __not_first_call = <optimized out>
        thr = <optimized out>
        link = 0x636b87467350
        start = 0x636b70627c90 <nxt_router_thread_start>
#4  0x00007a191c65c1f5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x00007a191c6dbb00 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

[ac000]

Thanks.

#0  nxt_h1p_complete_buffers (task=task@entry=0x7a1904001620, h1p=h1p@entry=0x636b874f99f0, all=all@entry=1) at src/nxt_h1proto.c:1522
        size = <optimized out>
        b = 0x0
        in = <optimized out>
        next = <optimized out>
        c = 0x0

c is NULL here which we're not expecting.

If you are able to, you could try this patch and see if anything else falls out...

diff --git ./src/nxt_h1proto.c ./src/nxt_h1proto.c
index 9a9ad553..d0e03077 100644
--- ./src/nxt_h1proto.c
+++ ./src/nxt_h1proto.c
@@ -1519,7 +1519,7 @@ nxt_h1p_complete_buffers(nxt_task_t *task, nxt_h1proto_t *h1p, nxt_bool_t all)
 
     b = h1p->buffers;
     c = h1p->conn;
-    in = c->read;
+    in = c ? c->read : NULL;
 
     if (b != NULL) {
         if (in == NULL) {

[ac000]

Just to confirm again.

If you hit the problematic page, then after about 3 minutes, Unit crashes?

It's only that one page that causes the crash?

If you leave the page loaded in the browse, does it survive?r

[Shaun-3adesign]

we make a call to the server this could come from a python process using python requests, but in our tests here we are using a browser. after making the call after about 3 minutes we get the crash the webpage looks fine and we can continue making requests, if we refresh the page we get another crash after 3 mins.

[Shaun-3adesign]

ok I tested the patch and still have the same issue.

latest backtrace

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `unit: router                                               '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  nxt_h1p_complete_buffers (task=task@entry=0x732064001620, h1p=h1p@entry=0x584bc2be89c0, all=all@entry=1) at src/nxt_h1proto.c:1522
1522    src/nxt_h1proto.c: No such file or directory.
[Current thread is 1 (Thread 0x73206b5b06c0 (LWP 297))]
(gdb) bt
#0  nxt_h1p_complete_buffers (task=task@entry=0x732064001620, h1p=h1p@entry=0x584bc2be89c0, all=all@entry=1) at src/nxt_h1proto.c:1522
#1  0x0000584ba231b46f in nxt_h1p_shutdown (task=0x732064001620, c=0x732064001550) at src/nxt_h1proto.c:2129
#2  0x0000584ba23075c2 in nxt_event_engine_start (engine=0x584bc2be8730) at src/nxt_event_engine.c:542
#3  0x0000584ba2305bf1 in nxt_thread_trampoline (data=0x584bc2b6a630) at src/nxt_thread.c:126
#4  0x000073206c3eb1f5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x000073206c46ab00 in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt full
#0  nxt_h1p_complete_buffers (task=task@entry=0x732064001620, h1p=h1p@entry=0x584bc2be89c0, all=all@entry=1) at src/nxt_h1proto.c:1522
        size = <optimized out>
        b = 0x0
        in = <optimized out>
        next = <optimized out>
        c = 0x0
#1  0x0000584ba231b46f in nxt_h1p_shutdown (task=0x732064001620, c=0x732064001550) at src/nxt_h1proto.c:2129
        timer = <optimized out>
        h1p = 0x584bc2be89c0
#2  0x0000584ba23075c2 in nxt_event_engine_start (engine=0x584bc2be8730) at src/nxt_event_engine.c:542
        obj = 0x7320640015b8
        data = 0x0
        task = 0x732064001620
        timeout = <optimized out>
        now = <optimized out>
        thr = <optimized out>
        handler = 0x584ba2307a80 <nxt_timer_handler>
#3  0x0000584ba2305bf1 in nxt_thread_trampoline (data=0x584bc2b6a630) at src/nxt_thread.c:126
        __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {126583077275328, 1042248611634940760, -1368, 11, 140720990601120, 126583068884992, -1714154970403239080, -4692315440823465128}, __mask_was_saved = 0}}, __pad = {0x73206b5afa30, 0x0, 0x0, 0x0}}
        __cancel_routine = 0x584ba2305a60 <nxt_thread_time_cleanup>
        __cancel_arg = <optimized out>
        __not_first_call = <optimized out>
        thr = <optimized out>
        link = 0x584bc2b6a630
        start = 0x584ba2310c90 <nxt_router_thread_start>
#4  0x000073206c3eb1f5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#5  0x000073206c46ab00 in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

[ac000]

Weird. Are you absolutely sure you tested the patched version?

With that patch, in this case, b & in should both then be NULL. We should then just skip past both the outer if()'s, however the key bit being that if c is NULL we won't attempt to read c->read.

Are websockets involved?