代码拉取完成，页面将自动刷新
更新失败，请稍后重试！
/
详情
AttributeError: None object has no attribute: size
评论 (1)
Vencen-31
创建了任务
原值
1. 问题说明：
我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码，导入了Gmlp模型。修改如图所示：
图 1 代码修改
2. 我的环境如下：
os:欧拉
mindspore:1.10.1
python：3.7
Ascend:910
cann：
3. 两次尝试的报错
1. 这里是列表文本在 **jupyter lab** 中通过命令：mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt
**得到以下报错**
```
----------------------------------------
Num devices: 2
Distributed mode: True
Num training samples: 1281168
Num classes: 1000
Num batches: 5005
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[28690,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
2. 这里是列表文本神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息
```
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
[WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again
Traceback (most recent call last):
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module>
train(args)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown
len(cache))
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[43151,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
如下图所示可以看到data and label 都不为None
图 2 debug 展示
菜鸡打昇腾AI比赛中... 先感谢各位大佬帮忙！
新值
1. 问题说明：
我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码，导入了Gmlp模型。修改如图所示：
图 1 代码修改
2. 我的环境如下：
os:欧拉
mindspore:1.10.1
python：3.7
Ascend:910
cann：
3. 两次尝试的报错
1. 这里是在 **jupyter lab** 中通过命令：mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt
**得到以下报错**
```
----------------------------------------
Num devices: 2
Distributed mode: True
Num training samples: 1281168
Num classes: 1000
Num batches: 5005
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[28690,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
2. 这里是列表文本神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息
```
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
[WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again
Traceback (most recent call last):
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module>
train(args)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown
len(cache))
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[43151,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
如下图所示可以看到data and label 都不为None
图 2 debug 展示
菜鸡打昇腾AI比赛中... 先感谢各位大佬帮忙！
原值
1. 问题说明：
我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码，导入了Gmlp模型。修改如图所示：
图 1 代码修改
2. 我的环境如下：
os:欧拉
mindspore:1.10.1
python：3.7
Ascend:910
cann：
3. 两次尝试的报错
1. 这里是在 **jupyter lab** 中通过命令：mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt
**得到以下报错**
```
----------------------------------------
Num devices: 2
Distributed mode: True
Num training samples: 1281168
Num classes: 1000
Num batches: 5005
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[28690,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
2. 这里是列表文本神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息
```
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
[WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again
Traceback (most recent call last):
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module>
train(args)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown
len(cache))
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[43151,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
如下图所示可以看到data and label 都不为None
图 2 debug 展示
菜鸡打昇腾AI比赛中... 先感谢各位大佬帮忙！
新值
1. 问题说明：
我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码，导入了Gmlp模型。修改如图所示：
图 1 代码修改
2. 我的环境如下：
os:欧拉
mindspore:1.10.1
python：3.7
Ascend:910
cann：
3. 两次尝试的报错
1. 这里是在 **jupyter lab** 中通过命令：mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt
**得到以下报错**
```
----------------------------------------
Num devices: 2
Distributed mode: True
Num training samples: 1281168
Num classes: 1000
Num batches: 5005
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[28690,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
2. 神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息
```
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
[WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again
Traceback (most recent call last):
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module>
train(args)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown
len(cache))
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[43151,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
如下图所示可以看到data and label 都不为None
图 2 debug 展示
菜鸡打昇腾AI比赛中... 先感谢各位大佬帮忙！
原值
1. 问题说明：
我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码，导入了Gmlp模型。修改如图所示：
图 1 代码修改
2. 我的环境如下：
os:欧拉
mindspore:1.10.1
python：3.7
Ascend:910
cann：
3. 两次尝试的报错
1. 这里是在 **jupyter lab** 中通过命令：mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt
**得到以下报错**
```
----------------------------------------
Num devices: 2
Distributed mode: True
Num training samples: 1281168
Num classes: 1000
Num batches: 5005
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[28690,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
2. 神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息
```
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
[WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again
Traceback (most recent call last):
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module>
train(args)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown
len(cache))
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[43151,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
如下图所示可以看到data and label 都不为None
图 2 debug 展示
菜鸡打昇腾AI比赛中... 先感谢各位大佬帮忙！
新值
1. 问题说明：
我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码，导入了Gmlp模型。修改如图所示：
图 1 代码修改
2. 在ModelArts中的环境如下：
os:欧拉
mindspore:1.10.1
python：3.7
Ascend:910
cann：
3. 两次尝试的报错
1. 这里是在 **jupyter lab** 中通过命令：mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt
**得到以下报错**
```
----------------------------------------
Num devices: 2
Distributed mode: True
Num training samples: 1281168
Num classes: 1000
Num batches: 5005
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[28690,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
2. 神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息
```
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
[WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again
Traceback (most recent call last):
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module>
train(args)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown
len(cache))
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[43151,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
如下图所示可以看到data and label 都不为None
图 2 debug 展示
菜鸡打昇腾AI比赛中... 先感谢各位大佬帮忙！
原值
1. 问题说明：
我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码，导入了Gmlp模型。修改如图所示：
图 1 代码修改
2. 在ModelArts中的环境如下：
os:欧拉
mindspore:1.10.1
python：3.7
Ascend:910
cann：
3. 两次尝试的报错
1. 这里是在 **jupyter lab** 中通过命令：mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt
**得到以下报错**
```
----------------------------------------
Num devices: 2
Distributed mode: True
Num training samples: 1281168
Num classes: 1000
Num batches: 5005
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[28690,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
2. 神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息
```
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
[WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again
Traceback (most recent call last):
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module>
train(args)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown
len(cache))
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[43151,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
如下图所示可以看到data and label 都不为None
图 2 debug 展示
菜鸡打昇腾AI比赛中... 先感谢各位大佬帮忙！
新值
1. 问题说明：
我基于mindcv仓库的[train_with_func.py](https://gitee.com/mindspore-lab/mindcv/blob/main/train_with_func.py)修改了代码，导入了Gmlp模型。修改如图所示：
图 1 代码修改
2. 在ModelArts中的环境如下：
os:欧拉
mindspore:1.10.1
python：3.7
Ascend:910
cann：
3. 两次尝试的报错
1. 这里是在 **jupyter lab** 中通过命令：mpirun --allow-run-as-root -n 2 python3 train_with_func.py --distribute &> log.txt
**得到以下报错**
```
----------------------------------------
Num devices: 2
Distributed mode: True
Num training samples: 1281168
Num classes: 1000
Num batches: 5005
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_1/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
Traceback (most recent call last):
File "train_with_func.py", line 506, in <module>
train(args)
File "train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/gmlp_mindspore/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[28690,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
2. 神奇的是在 **vscode** 中以相同的命令运行代码得到了更多貌似有用的报错信息
```
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
----------------------------------------
Num devices: 1
Distributed mode: False
Num training samples: 1281167
Num classes: 1000
Num batches: 10010
Batch size: 128
Auto augment: autoaug
Model: mobilenet_v2_035_224
Model param: 73075392
Num epochs: 300
Optimizer: adamw
LR: 0.001
LR Scheduler: cosine_decay
----------------------------------------
Start training
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 1 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=1, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=1[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 281, in main
old_handlers)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/forkserver.py", line 317, in _serve_one
code = spawn._main(child_r)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 24, in <module>
from gmlp import gMLPVision
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 115, in <module>
class SpatialGatingUnit(nn.Cell):
File "/home/ma-user/work/gmlp_mindspore/gmlp.py", line 121, in SpatialGatingUnit
act=nn.Identity(),
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/layer/basic.py", line 360, in __init__
super(Identity, self).__init__()
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/nn/cell.py", line 120, in __init__
init_pipeline()
RuntimeError: Device 0 call rtSetDevice failed, ret[507033].
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E39999: Inner Error, Please contact support engineer!
E39999 tsd client wait response fail, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:243]
TraceBack (most recent call last):
TsdOpen failed. devId=0, tdt error=31[FUNC:startAicpuExecutor][FILE:runtime.cc][LINE:1644]
Start aicpu executor failed, retCode=0x7020009 devId=0[FUNC:DeviceRetain][FILE:runtime.cc][LINE:2093]
Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1946]
Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:1973]
Check param failed, context can not be null.[FUNC:NewDevice][FILE:api_impl.cc][LINE:1315]
new device failed, retCode=0x7010006[FUNC:SetDevice][FILE:api_impl.cc][LINE:1336]
rtSetDevice execute failed, reason=[device retain error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_deprecated_interface.cc:293 OpenTsd
[WARNING] MD(581542,ffffb8a6ba40,python):2023-08-13-22:39:34.447.416 [mindspore/ccsrc/minddata/dataset/util/task.cc:163] Join] MapOp::WorkerEntry Thread ID 281467527623136 is not responding. Interrupt again
Traceback (most recent call last):
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 506, in <module>
train(args)
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 296, in train
log_interval=args.log_interval,
File "/home/ma-user/work/gmlp_mindspore/train_with_func.py", line 410, in train_epoch
loss, logits = train_step(data, label)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 594, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 98, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 405, in __call__
phase = self.compile(args_list, self.fn.__name__)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/common/api.py", line 379, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, phase, True)
AttributeError: None object has no attribute: size
----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
The function call stack (See file '/home/ma-user/work/rank_0/om/analyze_fail.dat' for more details. Get instructions about `analyze_fail.dat` at https://www.mindspore.cn/search?inputValue=analyze_fail.dat):
# 0 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:389
(loss, logits), grads = grad_fn(data, label)
^
# 1 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:554
return grad_(fn_, weights)(*args, **kwargs)
^
# 2 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/composite/base.py:498
outputs = fn(*args)
^
# 3 In file /home/ma-user/work/gmlp_mindspore/train_with_func.py:373
loss = loss_fn(logits, label)
^
# 4 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindcv/loss/cross_entropy_smooth.py:51
loss_logits = F.cross_entropy(
# 5 In file /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/mindspore/ops/function/nn_func.py:0
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/pipeline/jit/static_analysis/prim.cc:1613 GetEvaluatedValueForBuiltinTypeAttrOrMethod
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 28 leaked semaphores to clean up at shutdown
len(cache))
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[43151,1],1]
Exit code: 1
--------------------------------------------------------------------------
```
如下图所示可以看到data and label 都不为None
图 2 debug 展示
感谢各位大佬帮忙！
展开全部操作日志
折叠全部操作日志
误判申诉
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
取消
提交
登录 后才可以发表评论