[Bug]: Ascned上PYNATIVE_MODE执行报错，RuntimeError: Launch kernel failed, name:Default/MaxPool3DGradWithArgmax-op806

### 问题描述

在Ascend上PYNATIVE_MODE模式运行会报错："RuntimeError: Launch kernel failed, name:Default/MaxPool3DGradWithArgmax-op806",GRAPH_MODE模式可以正常执行。代码里其实也没用到MaxPool3D。

![网络结构](https://foruda.gitee.com/images/1715743980046864533/36f65db7_11527352.png "屏幕截图")
![网络结构](https://foruda.gitee.com/images/1715743995528249772/c4d3e878_11527352.png "屏幕截图")
![网络结构](https://foruda.gitee.com/images/1715744006067251053/76dc5bbe_11527352.png "屏幕截图")

### 环境信息

- **Hardware Environment(`Ascend`/`GPU`/`CPU`)  / 硬件环境**: 
> Please delete the backend not involved / 请删除不涉及的后端:
> /device Ascend

- **Software Environment / 软件环境 (Mandatory / 必填)**:
-- MindSpore version (e.g., 2.2.12) :
-- Python version (e.g., Python 3.7.16) :
-- OS platform and distribution (e.g., Linux ascend910 5.4.0-144-generic #161-Ubuntu SMP ):
-- GCC/Compiler version (if compiled from source):

- **Execute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: 
> Please delete the mode not involved / 请删除不涉及的模式:
> /mode pynative

### 关联用例

xxxxx

### 重现步骤

xxxx

### 预期结果

PYNATIVE_MODE运行不报错

### 日志/截图

[在这里上传图片]
(xsy_ms2.2) xidian@ascend910:/media/data1/xidian/wcx/MS_RelationNet$ /home/xidian/.conda/envs/xsy_ms2.2/bin/python /media/data1/xidian/wcx/MS_RelationNet/main_merge.py
[WARNING] ME(1323177:140247964157760,MainProcess):2024-05-15-03:08:39.420.900 [mindspore/run_check/_check_version.py:348] Using custom Ascend AI software package (Ascend Data Center Solution) path, package version checking is skipped. Please make sure Ascend AI software package (Ascend Data Center Solution) version is supported. For details, refer to the installation guidelines https://www.mindspore.cn/install
[WARNING] ME(1323177:140247964157760,MainProcess):2024-05-15-03:08:39.421.105 [mindspore/run_check/_check_version.py:468] Can not find driver so(need by mindspore-ascend). Please check whether the Environment Variable LD_LIBRARY_PATH is set. For details, refer to the installation guidelines: https://www.mindspore.cn/install
{}
{'BATCH_NUM_PER_CLASS': 19,
 'CLASS_NUM': 5,
 'EPISODE': 1000000,
 'FEATURE_DIM': 64,
 'HIDDEN_UNIT': 10,
 'LEARNING_RATE': 0.001,
 'PRINT_FREQUENCY': 20,
 'RELATION_DIM': 8,
 'SAMPLE_NUM_PER_CLASS': 1,
 'TEST_EPISODE': 200,
 'TEST_FREQUENCY': 40,
 'config_path': '/media/data1/xidian/wcx/MS_RelationNet/utils/../default_config.yaml',
 'encoder_checkpoint': 'omniglot_feature_encoder.ckpt',
 'model_root': 'checkpoints',
 'relation_checkpoint': 'omniglot_relation_network.ckpt'}
Please check the above information for the configurations
init data folders
init neural networks
Training...
[ERROR] KERNEL(1323177,7f8c8e7fc700,python):2024-05-15-03:14:21.061.152 [mindspore/ccsrc/plugin/device/ascend/kernel/acl/acl_kernel_mod.cc:253] Launch] Kernel launch failed, msg: Acl compile and execute failed, op_type_:CustMaxPool3DGradWithArgmax

----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EZ9999: Inner Error!
EZ9999  Attr data_format() only support NCDHW.[FUNC:MaxPool3DGradWithArgmaxVerify][FILE:nn_pooling_ops_proto.cc][LINE:514]
        TraceBack (most recent call last):
        Verifying CustMaxPool3DGradWithArgmax24 failed.[FUNC:InferShapeAndType][FILE:infershape_pass.cc][LINE:137]
        Sessin_id 0 does not exist, graph_id 23[FUNC:GetJsonObject][FILE:analyzer.cc][LINE:162]
        Param:graph_info is nullptr, check invalid[FUNC:DoAnalyze][FILE:analyzer.cc][LINE:260]
        Param:graph_info is nullptr, check invalid[FUNC:SaveAnalyzerDataToFile][FILE:analyzer.cc][LINE:217]
        Call InferShapeAndType for node:CustMaxPool3DGradWithArgmax24(CustMaxPool3DGradWithArgmax) failed[FUNC:Infer][FILE:infershape_pass.cc][LINE:119]
        process pass InferShapePass on node:CustMaxPool3DGradWithArgmax24 failed, ret:4294967295[FUNC:RunPassesOnNode][FILE:base_pass.cc][LINE:571]
        build graph failed, graph id:23, ret:1343242270[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
        [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 1343242270[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

----------------------------------------------------
- Ascend Warning Message:
----------------------------------------------------
W49999: If want to reuse binary file, please donwload binary file and install first![FUNC:BuildFusionOp][FILE:fusion_manager.cc][LINE:3763]
W11001: Op [Cast5] does not hit the high-priority operator information library, which might result in compromised performance.

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/transform/acl_ir/acl_utils.cc:406 Run

[ERROR] DEVICE(1323177,7f8c8e7fc700,python):2024-05-15-03:14:21.061.202 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_kernel_executor.cc:566] LaunchKernel] Launch kernel failed, kernel full name: Default/MaxPool3DGradWithArgmax-op806
Traceback (most recent call last):
  File "/media/data1/xidian/wcx/MS_RelationNet/main_merge.py", line 143, in <module>
    main()
  File "/media/data1/xidian/wcx/MS_RelationNet/main_merge.py", line 127, in main
    loss = train_one_step(folder=metatrain_character_folders,net=train_net)
  File "/media/data1/xidian/wcx/MS_RelationNet/core/train1.py", line 39, in train_one_step
    loss = net(cat_data,batch_labels)
  File "/home/xidian/.conda/envs/xsy_ms2.2/lib/python3.7/site-packages/mindspore/nn/cell.py", line 705, in __call__
    raise err
  File "/home/xidian/.conda/envs/xsy_ms2.2/lib/python3.7/site-packages/mindspore/nn/cell.py", line 701, in __call__
    output = self._run_construct(args, kwargs)
  File "/home/xidian/.conda/envs/xsy_ms2.2/lib/python3.7/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
    output = self.construct(*cast_inputs, **kwargs)
  File "/home/xidian/.conda/envs/xsy_ms2.2/lib/python3.7/site-packages/mindspore/nn/wrap/cell_wrapper.py", line 418, in construct
    return self._no_sens_impl(*inputs)
  File "/home/xidian/.conda/envs/xsy_ms2.2/lib/python3.7/site-packages/mindspore/nn/wrap/cell_wrapper.py", line 434, in _no_sens_impl
    grads = self.grad_no_sens(self.network, self.weights)(*inputs)
  File "/home/xidian/.conda/envs/xsy_ms2.2/lib/python3.7/site-packages/mindspore/ops/composite/base.py", line 388, in after_grad
    return grad_(fn, weights)(*args, **kwargs)
  File "/home/xidian/.conda/envs/xsy_ms2.2/lib/python3.7/site-packages/mindspore/common/api.py", line 121, in wrapper
    results = fn(*arg, **kwargs)
  File "/home/xidian/.conda/envs/xsy_ms2.2/lib/python3.7/site-packages/mindspore/ops/composite/base.py", line 378, in after_grad
    out = _pynative_executor()
  File "/home/xidian/.conda/envs/xsy_ms2.2/lib/python3.7/site-packages/mindspore/common/api.py", line 1147, in __call__
    return self._executor()
RuntimeError: Launch kernel failed, name:Default/MaxPool3DGradWithArgmax-op806

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

Error in atexit._run_exitfuncs:
RuntimeError: Call aclrtDestroyStream failed, ret = 0x107003

----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EE9999: Inner Error!
EE9999  Stream destroy failed, stream is not in current ctx.[FUNC:StreamDestroy][FILE:api_impl.cc][LINE:874]
        TraceBack (most recent call last):
        rtStreamDestroy execute failed, reason=[stream not in current context][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
        destroy stream failed, runtime result = 107003[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
EZ9999: Inner Error!
EZ9999  Attr data_format() only support NCDHW.[FUNC:MaxPool3DGradWithArgmaxVerify][FILE:nn_pooling_ops_proto.cc][LINE:514]
        TraceBack (most recent call last):
        Verifying CustMaxPool3DGradWithArgmax24 failed.[FUNC:InferShapeAndType][FILE:infershape_pass.cc][LINE:137]
        Sessin_id 0 does not exist, graph_id 23[FUNC:GetJsonObject][FILE:analyzer.cc][LINE:162]
        Param:graph_info is nullptr, check invalid[FUNC:DoAnalyze][FILE:analyzer.cc][LINE:260]
        Param:graph_info is nullptr, check invalid[FUNC:SaveAnalyzerDataToFile][FILE:analyzer.cc][LINE:217]
        Call InferShapeAndType for node:CustMaxPool3DGradWithArgmax24(CustMaxPool3DGradWithArgmax) failed[FUNC:Infer][FILE:infershape_pass.cc][LINE:119]
        process pass InferShapePass on node:CustMaxPool3DGradWithArgmax24 failed, ret:4294967295[FUNC:RunPassesOnNode][FILE:base_pass.cc][LINE:571]
        build graph failed, graph id:23, ret:1343242270[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
        [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 1343242270[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

Aborted (core dumped)
(xsy_ms2.2) xidian@ascend910:/media/data1/xidian/wcx/MS_RelationNet$

MindSpore/models

内容风险标识

评论 (10)