Skip to content

单机8块P800运行Qwen3-235B-A22B总是loadlayers开始的时候崩溃, xpu_ipc_get_memhandle failed 。 #8102

Description

@zziawan

INFO 2026-07-16 14:43:22,869 1748 engine.py[line:151] Waiting for worker processes to be ready...
Loading Weights: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:43<00:00, 1.04s/it]
Loading Layers: 0%| | 0/100 [00:11<?, ?it/s]
ERROR 2026-07-16 14:45:22,612 1748 engine.py[line:160] Failed to launch worker processes, check log/workerlog.* for more details.

ERROR 2026-07-16 14:45:31,391 1748 engine.py[line:447] Error extracting sub services: [Errno 3] No such process, Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/engine.py", line 444, in _exit_sub_services
pgid = os.getpgid(self.worker_proc.pid) Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/../worker/worker_process.py", line 1296, in
run_worker_proc()
File "/usr/local/lib/python3.10/dist-packages/paddle/base/dygraph/base.py", line 406, in _decorate_function
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/../worker/worker_process.py", line 1281, in run_worker_proc
worker_proc.initialize_kv_cache()
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/engine/../worker/worker_process.py", line 659, in initialize_kv_cache
available_kv_cache_memory = self.worker.determine_available_memory()
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/worker/xpu_worker.py", line 130, in determine_available_memory
self.model_runner.profile_run()
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/worker/xpu_model_runner.py", line 1711, in profile_run
self.initialize_kv_cache(profile=True)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/worker/xpu_model_runner.py", line 1270, in initialize_kv_cache
set_data_ipc(key_cache, key_cache_name)
File "/usr/local/lib/python3.10/dist-packages/fastdeploy/model_executor/ops/xpu/fastdeploy_ops/init.py", line 2000, in set_data_ipc
outs = _C_ops._run_custom_op("set_data_ipc", tmp_input,shm_name)
OSError: xpu_ipc_get_memhandle failed

使用的是fastdeploy 2.5镜像,启动参数及模型启动参数均参考了文档的参数。请问有没有哪位碰到过类似问题。

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions