The algorithm failed on one or more cases.

The algorithm failed on one or more cases. ¶

By: COOl on July 28, 2023, 12:05 p.m.

Dear organizer, Could you please tell me the error on the submission on July 28, 2023, 7:39 p.m.? Meantime, the same docker file tested on Try-out for Algorithm using data from training set is successful. A comprehensive reply is very appreciated. Thank you so much!

Best, COOl

Last edited by: COOl on Aug. 15, 2023, 12:58 p.m., edited 2 times in total.

Re: The algorithm failed on one or more cases. ¶

By: giansteve on July 31, 2023, 8:05 a.m.

Dear COOI,

we found a lot of submissions from you on that date. Unfortunately, the submission time is different for each user depending on their time zone. Therefore, I will paste here all your failed submission's logs I can find on that date. For the future, please provide the submission ID to facilitate its identification.

ID: 079c0377-348c-46f9-a24b-6ce06bb05344 error: Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. stderr:

2023-07-28T09:13:43.576000+00:00 Traceback (most recent call last):
2023-07-28T09:13:43.576000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 470, in predict_logits_from_preprocessed_data
2023-07-28T09:13:43.577000+00:00     prediction = self.predict_sliding_window_return_logits(data)
2023-07-28T09:13:43.577000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 569, in predict_sliding_window_return_logits
2023-07-28T09:13:43.577000+00:00     self.network = self.network.to(self.device)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to
2023-07-28T09:13:43.577000+00:00     return self._apply(convert)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.577000+00:00     module._apply(fn)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.577000+00:00     module._apply(fn)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.577000+00:00     module._apply(fn)
2023-07-28T09:13:43.577000+00:00   [Previous line repeated 4 more times]
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
2023-07-28T09:13:43.577000+00:00     param_applied = fn(param)
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert
2023-07-28T09:13:43.577000+00:00     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2023-07-28T09:13:43.577000+00:00 RuntimeError: CUDA error: invalid device ordinal
2023-07-28T09:13:43.577000+00:00 CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2023-07-28T09:13:43.577000+00:00 For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2023-07-28T09:13:43.577000+00:00 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2023-07-28T09:13:43.577000+00:00 
2023-07-28T09:13:43.577000+00:00 Traceback (most recent call last):
2023-07-28T09:13:43.577000+00:00   File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2023-07-28T09:13:43.577000+00:00     return _run_code(code, main_globals, None,
2023-07-28T09:13:43.577000+00:00   File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
2023-07-28T09:13:43.577000+00:00     exec(code, run_globals)
2023-07-28T09:13:43.577000+00:00   File "/opt/app/process.py", line 112, in <module>
2023-07-28T09:13:43.577000+00:00     Segaalgorithm(args).process()
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/evalutils/evalutils.py", line 183, in process
2023-07-28T09:13:43.577000+00:00     self.process_cases()
2023-07-28T09:13:43.577000+00:00   File "/home/user/.local/lib/python3.8/site-packages/evalutils/evalutils.py", line 191, in process_cases
2023-07-28T09:13:43.577000+00:00     self._case_results.append(self.process_case(idx=idx, case=case))
2023-07-28T09:13:43.577000+00:00   File "/opt/app/process.py", line 60, in process_case
2023-07-28T09:13:43.577000+00:00     predictions = self.predict(input_image=input_image)
2023-07-28T09:13:43.578000+00:00   File "/opt/app/process.py", line 99, in predict
2023-07-28T09:13:43.578000+00:00     ret = self.predictor.predict_single_npy_array(img, props, None, None, False)
2023-07-28T09:13:43.578000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 425, in predict_single_npy_array
2023-07-28T09:13:43.578000+00:00     predicted_logits = self.predict_logits_from_preprocessed_data(dct['data']).cpu()
2023-07-28T09:13:43.578000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 494, in predict_logits_from_preprocessed_data
2023-07-28T09:13:43.578000+00:00     prediction = self.predict_sliding_window_return_logits(data)
2023-07-28T09:13:43.578000+00:00   File "/opt/app/nnunetv2/inference/predict_from_raw_data.py", line 569, in predict_sliding_window_return_logits
2023-07-28T09:13:43.578000+00:00     self.network = self.network.to(self.device)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to
2023-07-28T09:13:43.578000+00:00     return self._apply(convert)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.578000+00:00     module._apply(fn)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.578000+00:00     module._apply(fn)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
2023-07-28T09:13:43.578000+00:00     module._apply(fn)
2023-07-28T09:13:43.578000+00:00   [Previous line repeated 4 more times]
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
2023-07-28T09:13:43.578000+00:00     param_applied = fn(param)
2023-07-28T09:13:43.578000+00:00   File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert
2023-07-28T09:13:43.578000+00:00     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2023-07-28T09:13:43.578000+00:00 RuntimeError: CUDA error: invalid device ordinal
2023-07-28T09:13:43.578000+00:00 CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
2023-07-28T09:13:43.578000+00:00 For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2023-07-28T09:13:43.578000+00:00 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2023-07-28T09:13:43.578000+00:00

ID: f161a705-1867-45b2-a402-10d70eb63643 error: 100%|██████████| 245/245 [03:43<00:00, 1.10it/s] stdout:

2023-07-28T11:46:00.901000+00:00 nnUNet_raw is not defined and nnU-Net can only be used on data for which preprocessed files are already present on your system. nnU-Net cannot be used for experiment planning and preprocessing like this. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up properly.
2023-07-28T11:46:00.901000+00:00 nnUNet_preprocessed is not defined and nnU-Net can not be used for preprocessing or training. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up.
2023-07-28T11:46:00.902000+00:00 nnUNet_results is not defined and nnU-Net cannot be used for training or inference. If this is not intended behavior, please read documentation/setting_up_paths.md for information on how to set this up.

ID: fbef9b05-5a60-4ca3-8754-7cb687a86108 error: 100%|██████████| 245/245 [03:40<00:00, 1.11it/s] stdout:

2023-07-28T17:26:25.075000+00:00 nnUNet_raw is not defined and nnU-Net can only be used on data for which preprocessed files are already present on your system. nnU-Net cannot be used for experiment planning and preprocessing like this. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up properly.
2023-07-28T17:26:25.075000+00:00 nnUNet_preprocessed is not defined and nnU-Net can not be used for preprocessing or training. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up.
2023-07-28T17:26:25.075000+00:00 nnUNet_results is not defined and nnU-Net cannot be used for training or inference. If this is not intended behavior, please read documentation/setting_up_paths.md for information on how to set this up.

I hope it helps Best Gian Marco

Re: The algorithm failed on one or more cases. ¶

By: COOl on July 31, 2023, 8:10 a.m.

Thanks for your kind reply!

Re: The algorithm failed on one or more cases. ¶

By: COOl on July 31, 2023, 8:31 a.m.

Dear organizer, Can you help me with the error occuring in id: 7f83b183-5cdd-4be2-a3c1-87cedb4d6a79? Thank you