SEG.A. - Segmentation of the Aorta Banner
Join

The algorithm failed on one or more cases.

The algorithm failed on one or more cases.  

  By: COOl on July 28, 2023, 9:26 a.m.

Could you please tell me the error on the submission on July 28, 2023, 7:39 p.m.? Thank you so much!

 Last edited by: COOl on Aug. 15, 2023, 12:58 p.m., edited 1 time in total.
COOl
COOl
Posts: 9

Re: The algorithm failed on one or more cases.  

  By: giansteve on July 28, 2023, 9:48 a.m.

Hi,

your last submission shows the error "Compile with TORCH_USE_CUDA_DSA to enable device-side assertions."

with quite some tracebacks:

2023-07-28T09:13:43.576000+00:00 Traceback (most recent call last):
2023-07-28T09:13:43.576000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 470, in predict_logits_from_preprocessed_data
2023-07-28T09:13:43.577000+00:00     prediction = self.predict_sliding_window_return_logits(data)
2023-07-28T09:13:43.577000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 569, in predict_sliding_window_return_logits
2023-07-28T09:13:43.577000+00:00     self.network = self.network.to(self.device)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to
2023-07-28T09:13:43.577000+00:00     return self._apply(convert)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.577000+00:00     module._apply(fn)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.577000+00:00     module._apply(fn)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.577000+00:00     module._apply(fn)
2023-07-28T09:13:43.577000+00:00   [Previous line repeated 4 more times]
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
2023-07-28T09:13:43.577000+00:00     param_applied = fn(param)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert
2023-07-28T09:13:43.577000+00:00     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2023-07-28T09:13:43.577000+00:00 RuntimeError: CUDA error: invalid device ordinal
2023-07-28T09:13:43.577000+00:00 CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2023-07-28T09:13:43.577000+00:00 For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2023-07-28T09:13:43.577000+00:00 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2023-07-28T09:13:43.577000+00:00 
2023-07-28T09:13:43.577000+00:00 Traceback (most recent call last):
2023-07-28T09:13:43.577000+00:00   File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2023-07-28T09:13:43.577000+00:00     return _run_code(code, main_globals, None,
2023-07-28T09:13:43.577000+00:00   File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
2023-07-28T09:13:43.577000+00:00     exec(code, run_globals)
2023-07-28T09:13:43.577000+00:00   File "/opt/app/process.py", line 112, in <module>
2023-07-28T09:13:43.577000+00:00     Segaalgorithm(args).process()
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/evalutils/evalutils.py", line 183, in process
2023-07-28T09:13:43.577000+00:00     self.process_cases()
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/evalutils/evalutils.py", line 191, in process_cases
2023-07-28T09:13:43.577000+00:00     self._case_results.append(self.process_case(idx=idx, case=case))
2023-07-28T09:13:43.577000+00:00   File "/opt/app/process.py", line 60, in process_case
2023-07-28T09:13:43.577000+00:00     predictions = self.predict(input_image=input_image)
2023-07-28T09:13:43.578000+00:00   File "/opt/app/process.py", line 99, in predict
2023-07-28T09:13:43.578000+00:00     ret = self.predictor.predict_single_npy_array(img, props, None, None, False)
2023-07-28T09:13:43.578000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 425, in predict_single_npy_array
2023-07-28T09:13:43.578000+00:00     predicted_logits = self.predict_logits_from_preprocessed_data(dct['data']).cpu()
2023-07-28T09:13:43.578000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 494, in predict_logits_from_preprocessed_data
2023-07-28T09:13:43.578000+00:00     prediction = self.predict_sliding_window_return_logits(data)
2023-07-28T09:13:43.578000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 569, in predict_sliding_window_return_logits
2023-07-28T09:13:43.578000+00:00     self.network = self.network.to(self.device)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to
2023-07-28T09:13:43.578000+00:00     return self._apply(convert)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.578000+00:00     module._apply(fn)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.578000+00:00     module._apply(fn)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.578000+00:00     module._apply(fn)
2023-07-28T09:13:43.578000+00:00   [Previous line repeated 4 more times]
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
2023-07-28T09:13:43.578000+00:00     param_applied = fn(param)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert
2023-07-28T09:13:43.578000+00:00     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2023-07-28T09:13:43.578000+00:00 RuntimeError: CUDA error: invalid device ordinal
2023-07-28T09:13:43.578000+00:00 CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2023-07-28T09:13:43.578000+00:00 For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2023-07-28T09:13:43.578000+00:00 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2023-07-28T09:13:43.578000+00:00
giansteve
giansteve
Posts: 45