The algorithm failed on one or more cases

Re: The algorithm failed on one or more cases ¶

By: Marcel.Frueh on Sept. 22, 2023, 6:50 a.m.

Your error only occurred within some images. One volume that led to an error was of shape (350x350x287)

Log:

2023-09-22T05:35:04.031000+00:00 creating1111111111111111
2023-09-22T05:35:04.031000+00:00 Checking GPU availability
2023-09-22T05:35:05.032000+00:00 Available: True
2023-09-22T05:35:05.032000+00:00 Device count: 1
2023-09-22T05:35:05.032000+00:00 Current device: 0
2023-09-22T05:35:05.032000+00:00 Device name: Tesla T4
2023-09-22T05:35:05.032000+00:00 Device memory: 15843721216

Error:

2023-09-22T05:35:31.038000+00:00 Traceback (most recent call last):
2023-09-22T05:35:31.038000+00:00   File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2023-09-22T05:35:31.038000+00:00     return _run_code(code, main_globals, None,
2023-09-22T05:35:31.038000+00:00   File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
2023-09-22T05:35:31.038000+00:00     exec(code, run_globals)
2023-09-22T05:35:31.038000+00:00   File "/opt/algorithm/process.py", line 77, in <module>
2023-09-22T05:35:31.038000+00:00     Petctsegmentationcontainer().process()
2023-09-22T05:35:31.038000+00:00   File "/opt/algorithm/process.py", line 71, in process
2023-09-22T05:35:31.038000+00:00     predict(self.model_path, self.nii_path, self.output_path)
2023-09-22T05:35:31.038000+00:00   File "/opt/algorithm/UNet.py", line 89, in predict
2023-09-22T05:35:31.038000+00:00     net.inference(model_path, output_path)
2023-09-22T05:35:31.038000+00:00   File "/opt/algorithm/UNet.py", line 59, in inference
2023-09-22T05:35:31.038000+00:00     sr_400 = np.zeros([batch_now, ori_size, ori_size])
2023-09-22T05:35:31.038000+00:00 TypeError: only integer tensors of a single element can be converted to an index

Last edited by: Marcel.Frueh on Sept. 22, 2023, 6:50 a.m., edited 1 time in total.

Re: The algorithm failed on one or more cases ¶

By: Sepideh on Sept. 22, 2023, 6:56 a.m.

Dear Marcel.

Could you please send me the latest error of my final? Thanks in advance

Re: The algorithm failed on one or more cases ¶

By: Marcel.Frueh on Sept. 22, 2023, 7:11 a.m.

For only a few images your received this error: the volumes were of shape 400x400x400

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.86 GiB (GPU 0; 14.76 GiB total capacity; 8.16 GiB already allocated; 2.08 GiB free; 11.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Last edited by: Marcel.Frueh on Sept. 22, 2023, 7:12 a.m., edited 1 time in total.

Re: The algorithm failed on one or more cases ¶

By: shadab on Sept. 22, 2023, 9:29 a.m.

Hi, could you please let me know my latest error log?

Re: The algorithm failed on one or more cases ¶

By: Marcel.Frueh on Sept. 22, 2023, 10:18 a.m.

Unexpected key(s) in state_dict: "model.0.conv.unit0.conv.weight", "model.0.conv.unit0.conv.bias", "model.0.conv.unit0.adn.N.weight", "model.0.conv.unit0.adn.N.bias", "model.0.conv.unit0.adn.N.running_mean", "model.0.conv.unit0.adn.N.running_var", "model.0.conv.unit0.adn.N.num_batches_tracked", "model.0.conv.unit0.adn.A.weight", "model.0.conv.unit1.conv.weight", "model.0.conv.unit1.conv.bias", "model.0.conv.unit1.adn.N.weight", "model.0.conv.unit1.adn.N.bias", "model.0.conv.unit1.adn.N.running_mean", "model.0.conv.unit1.adn.N.running_var", "model.0.conv.unit1.adn.N.num_batches_tracked", "model.0.conv.unit1.adn.A.weight", "model.0.residual.weight", "model.0.residual.bias", "model.1.submodule.0.conv.unit0.conv.weight", "model.1.submodule.0.conv.unit0.conv.bias", "model.1.submodule.0.conv.unit0.adn.N.weight", "model.1.submodule.0.conv.unit0.adn.N.bias", "model.1.submodule.0.conv.unit0.adn.N.running_mean", "model.1.submodule.0.conv.unit0.adn.N.running_var", "model.1.submodule.0.conv.unit0.adn.N.num_batches_tracke

Re: The algorithm failed on one or more cases ¶

By: anissa218 on Sept. 23, 2023, 8:08 a.m.

Hi Marcel,

Could you please send the error message on my latest submission to the final set?

Thanks!

Re: The algorithm failed on one or more cases ¶

By: Marcel.Frueh on Sept. 23, 2023, 10:34 a.m.

Log:

2023-09-22T22:51:44.875000+00:00 imports successful
2023-09-22T22:51:44.875000+00:00 START
2023-09-22T22:51:44.875000+00:00 Start prediction
2023-09-22T22:51:44.875000+00:00 nnUNet segmentation starting!
2023-09-22T22:51:47.876000+00:00 preprocessing
2023-09-22T22:53:02.901000+00:00 old shape: (347, 256, 256), new_shape: [578 400 400], old_spacing: [5.0, 3.1819000244140625, 3.1819000244140625], new_spacing: [3.0, 2.0364201068878174, 2.0364201068878174], fn_data: functools.partial(<function resample_data_or_seg_to_shape at 0x7fe00d04c280>, is_seg=False, order=3, order_z=0, force_separate_z=None)
2023-09-22T22:53:02.902000+00:00 predicting
2023-09-22T22:53:02.902000+00:00 Prediction with perform_everything_on_gpu=True failed due to insufficient GPU memory. Falling back to perform_everything_on_gpu=False. Not a big deal, just slower...
2023-09-22T22:53:02.902000+00:00 Error:

Error:

2023-09-22T22:53:02.902000+00:00 Traceback (most recent call last):
2023-09-22T22:53:02.902000+00:00   File "/opt/algorithm/nnUNet/nnunetv2/inference/predict_from_raw_data.py", line 468, in predict_logits_from_preprocessed_data
2023-09-22T22:53:02.902000+00:00     self.network.load_state_dict(params)
2023-09-22T22:53:02.902000+00:00   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
2023-09-22T22:53:02.902000+00:00     raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
2023-09-22T22:53:02.902000+00:00 RuntimeError: Error(s) in loading state_dict for PlainConvUNet:
2023-09-22T22:53:02.902000+00:00    size mismatch for decoder.transpconvs.0.weight: copying a param with shape torch.Size([320, 320, 1, 2, 2]) from checkpoint, the shape in current model is torch.Size([320, 320, 2, 2, 2]).
2023-09-22T22:53:02.902000+00:00 Traceback (most recent call last):
2023-09-22T22:53:02.902000+00:00   File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
2023-09-22T22:53:02.902000+00:00     return _run_code(code, main_globals, None,
2023-09-22T22:53:02.902000+00:00   File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
2023-09-22T22:53:02.902000+00:00     exec(code, run_globals)
2023-09-22T22:53:02.902000+00:00   File "/opt/algorithm/process.py", line 69, in <module>
2023-09-22T22:53:02.902000+00:00     Autopet_baseline().process()
2023-09-22T22:53:02.902000+00:00   File "/opt/algorithm/process.py", line 63, in process
2023-09-22T22:53:02.902000+00:00     self.predict()
2023-09-22T22:53:02.902000+00:00   File "/opt/algorithm/process.py", line 52, in predict
2023-09-22T22:53:02.902000+00:00     predictor.predict_single_npy_array(images, properties, None, output_file_trunc, False)
2023-09-22T22:53:02.902000+00:00   File "/opt/algorithm/nnUNet/nnunetv2/inference/predict_from_raw_data.py", line 428, in predict_single_npy_array
2023-09-22T22:53:02.902000+00:00     predicted_logits = self.predict_logits_from_preprocessed_data(dct['data']).cpu()
2023-09-22T22:53:02.902000+00:00   File "/opt/algorithm/nnUNet/nnunetv2/inference/predict_from_raw_data.py", line 492, in predict_logits_from_preprocessed_data
2023-09-22T22:53:02.902000+00:00     self.network.load_state_dict(params)
2023-09-22T22:53:02.903000+00:00   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
2023-09-22T22:53:02.903000+00:00     raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
2023-09-22T22:53:02.903000+00:00 RuntimeError: Error(s) in loading state_dict for PlainConvUNet:
2023-09-22T22:53:02.903000+00:00    size mismatch for decoder.transpconvs.0.weight: copying a param with shape torch.Size([320, 320, 1, 2, 2]) from checkpoint, the shape in current model is torch.Size([320, 320, 2, 2, 2]).

Last edited by: Marcel.Frueh on Sept. 23, 2023, 10:34 a.m., edited 1 time in total.

Re: The algorithm failed on one or more cases ¶

By: shadab on Sept. 24, 2023, 5:07 a.m.

Hi, I just submitted to the final test phase. My algorithm passed the prelim test phase but failed in the final phase. Could you let me know my detailed error log please? Thanks

Re: The algorithm failed on one or more cases ¶

By: Marcel.Frueh on Sept. 24, 2023, 7 a.m.

The error only occurred within some images, not all Log:

2023-09-24T05:03:28.008000+00:00 Checking GPU availability
2023-09-24T05:03:29.008000+00:00 Available: True
2023-09-24T05:03:29.008000+00:00 Device count: 1
2023-09-24T05:03:29.008000+00:00 Current device: 0
2023-09-24T05:03:29.008000+00:00 Device name: Tesla T4
2023-09-24T05:03:29.008000+00:00 Device memory: 15843721216
2023-09-24T05:03:29.008000+00:00 Start processing
2023-09-24T05:03:34.010000+00:00 Start prediction
2023-09-24T05:03:34.010000+00:00 starting

Error:

2023-09-24T05:03:37.012000+00:00 /home/algorithm/.local/lib/python3.10/site-packages/monai/utils/deprecate_utils.py:321: FutureWarning: monai.transforms.io.dictionary LoadImaged.__init__:image_only: Current default value of argument `image_only=False` has been deprecated since version 1.1. It will be changed to `image_only=True` in version 1.3.
2023-09-24T05:03:37.012000+00:00   warn_deprecated(argname, msg, warning_category)
2023-09-24T05:04:54.033000+00:00 Current run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x7f0d7049b280>
2023-09-24T05:04:54.033000+00:00 Engine run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x7f0d7049b280>
2023-09-24T05:04:54.033000+00:00 Traceback (most recent call last):
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
2023-09-24T05:04:54.033000+00:00     return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
2023-09-24T05:04:54.033000+00:00     return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/post/dictionary.py", line 436, in __call__
2023-09-24T05:04:54.033000+00:00     d[self.output_key] = self.ensemble(items)
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/post/array.py", line 635, in __call__
2023-09-24T05:04:54.033000+00:00     img_ = img_ * weights / weights.mean(dim=0, keepdim=True)
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/data/meta_tensor.py", line 276, in __torch_function__
2023-09-24T05:04:54.033000+00:00     ret = super().__torch_function__(func, types, args, kwargs)
2023-09-24T05:04:54.033000+00:00   File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 1295, in __torch_function__
2023-09-24T05:04:54.033000+00:00     ret = func(*args, **kwargs)
2023-09-24T05:04:54.033000+00:00 torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.37 GiB (GPU 0; 14.76 GiB total capacity; 11.14 GiB already allocated; 1.82 GiB free; 12.01 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-24T05:04:54.033000+00:00 
2023-09-24T05:04:54.033000+00:00 The above exception was the direct cause of the following exception:
2023-09-24T05:04:54.033000+00:00 
2023-09-24T05:04:54.033000+00:00 Traceback (most recent call last):
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
2023-09-24T05:04:54.033000+00:00     return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
2023-09-24T05:04:54.033000+00:00     return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/compose.py", line 322, in __call__
2023-09-24T05:04:54.033000+00:00     result = execute_compose(
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/compose.py", line 111, in execute_compose
2023-09-24T05:04:54.033000+00:00     data = apply_transform(
2023-09-24T05:04:54.033000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
2023-09-24T05:04:54.034000+00:00     raise RuntimeError(f"applying transform {transform}") from e
2023-09-24T05:04:54.034000+00:00 RuntimeError: applying transform <monai.transforms.post.dictionary.MeanEnsembled object at 0x7f0d7049a5c0>
2023-09-24T05:04:54.034000+00:00 
2023-09-24T05:04:54.034000+00:00 The above exception was the direct cause of the following exception:
2023-09-24T05:04:54.034000+00:00 
2023-09-24T05:04:54.034000+00:00 Traceback (most recent call last):
2023-09-24T05:04:54.034000+00:00   File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
2023-09-24T05:04:54.034000+00:00     return _run_code(code, main_globals, None,
2023-09-24T05:04:54.034000+00:00   File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
2023-09-24T05:04:54.034000+00:00     exec(code, run_globals)
2023-09-24T05:04:54.034000+00:00   File "/opt/algorithm/process.py", line 102, in <module>
2023-09-24T05:04:54.034000+00:00     Unet_baseline().process()
2023-09-24T05:04:54.034000+00:00   File "/opt/algorithm/process.py", line 88, in process
2023-09-24T05:04:54.034000+00:00     monai_unet.run_inference(
2023-09-24T05:04:54.034000+00:00   File "/opt/algorithm/monai_unet.py", line 176, in run_inference
2023-09-24T05:04:54.034000+00:00     segment_PETCT(
2023-09-24T05:04:54.034000+00:00   File "/opt/algorithm/monai_unet.py", line 157, in segment_PETCT
2023-09-24T05:04:54.034000+00:00     ensemble_evaluate(post_transforms, models, device, val_dataloader)
2023-09-24T05:04:54.034000+00:00   File "/opt/algorithm/monai_unet.py", line 130, in ensemble_evaluate
2023-09-24T05:04:54.034000+00:00     evaluator.run()
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/evaluator.py", line 148, in run
2023-09-24T05:04:54.034000+00:00     super().run()
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/workflow.py", line 283, in run
2023-09-24T05:04:54.034000+00:00     super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 892, in run
2023-09-24T05:04:54.034000+00:00     return self._internal_run()
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 935, in _internal_run
2023-09-24T05:04:54.034000+00:00     return next(self._internal_run_generator)
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 993, in _internal_run_as_gen
2023-09-24T05:04:54.034000+00:00     self._handle_exception(e)
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 638, in _handle_exception
2023-09-24T05:04:54.034000+00:00     raise e
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 959, in _internal_run_as_gen
2023-09-24T05:04:54.034000+00:00     epoch_time_taken += yield from self._run_once_on_dataset_as_gen()
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 1087, in _run_once_on_dataset_as_gen
2023-09-24T05:04:54.034000+00:00     self._handle_exception(e)
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 638, in _handle_exception
2023-09-24T05:04:54.034000+00:00     raise e
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 1068, in _run_once_on_dataset_as_gen
2023-09-24T05:04:54.034000+00:00     self.state.output = self._process_function(self, self.state.batch)
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/evaluator.py", line 462, in _iteration
2023-09-24T05:04:54.034000+00:00     engine.fire_event(IterationEvents.MODEL_COMPLETED)
2023-09-24T05:04:54.034000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 449, in fire_event
2023-09-24T05:04:54.034000+00:00     return self._fire_event(event_name)
2023-09-24T05:04:54.035000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 425, in _fire_event
2023-09-24T05:04:54.035000+00:00     func(*first, *(event_args + others), **kwargs)
2023-09-24T05:04:54.035000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/workflow.py", line 226, in _run_postprocessing
2023-09-24T05:04:54.035000+00:00     engine.state.batch[i], engine.state.output[i] = engine_apply_transform(b, o, posttrans)
2023-09-24T05:04:54.035000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/utils.py", line 261, in engine_apply_transform
2023-09-24T05:04:54.035000+00:00     transformed_data = apply_transform(transform, data)
2023-09-24T05:04:54.035000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
2023-09-24T05:04:54.035000+00:00     raise RuntimeError(f"applying transform {transform}") from e
2023-09-24T05:04:54.035000+00:00 RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f0d7049b280>

Re: The algorithm failed on one or more cases ¶

By: shadab on Sept. 24, 2023, 1:12 p.m.

Hi Marcel,

Thank you.

I tried to reduce the inference batch size but it seems to have failed again. Would it be possible to get the error again?

Re: The algorithm failed on one or more cases ¶

By: Marcel.Frueh on Sept. 24, 2023, 1:44 p.m.

2023-09-24T12:59:55.363000+00:00 /home/algorithm/.local/lib/python3.10/site-packages/monai/utils/deprecate_utils.py:321: FutureWarning: monai.transforms.io.dictionary LoadImaged.__init__:image_only: Current default value of argument `image_only=False` has been deprecated since version 1.1. It will be changed to `image_only=True` in version 1.3.
2023-09-24T12:59:55.364000+00:00   warn_deprecated(argname, msg, warning_category)
2023-09-24T13:01:13.382000+00:00 Current run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x7f661c5ab280>
2023-09-24T13:01:13.382000+00:00 Engine run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x7f661c5ab280>
2023-09-24T13:01:13.383000+00:00 Traceback (most recent call last):
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
2023-09-24T13:01:13.383000+00:00     return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
2023-09-24T13:01:13.383000+00:00     return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/post/dictionary.py", line 436, in __call__
2023-09-24T13:01:13.383000+00:00     d[self.output_key] = self.ensemble(items)
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/post/array.py", line 635, in __call__
2023-09-24T13:01:13.383000+00:00     img_ = img_ * weights / weights.mean(dim=0, keepdim=True)
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/data/meta_tensor.py", line 276, in __torch_function__
2023-09-24T13:01:13.383000+00:00     ret = super().__torch_function__(func, types, args, kwargs)
2023-09-24T13:01:13.383000+00:00   File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 1295, in __torch_function__
2023-09-24T13:01:13.383000+00:00     ret = func(*args, **kwargs)
2023-09-24T13:01:13.383000+00:00 torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.37 GiB (GPU 0; 14.76 GiB total capacity; 11.14 GiB already allocated; 2.67 GiB free; 11.16 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-24T13:01:13.383000+00:00 
2023-09-24T13:01:13.383000+00:00 The above exception was the direct cause of the following exception:
2023-09-24T13:01:13.383000+00:00 
2023-09-24T13:01:13.383000+00:00 Traceback (most recent call last):
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
2023-09-24T13:01:13.383000+00:00     return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
2023-09-24T13:01:13.383000+00:00     return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/compose.py", line 322, in __call__
2023-09-24T13:01:13.383000+00:00     result = execute_compose(
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/compose.py", line 111, in execute_compose
2023-09-24T13:01:13.383000+00:00     data = apply_transform(
2023-09-24T13:01:13.383000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
2023-09-24T13:01:13.383000+00:00     raise RuntimeError(f"applying transform {transform}") from e
2023-09-24T13:01:13.383000+00:00 RuntimeError: applying transform <monai.transforms.post.dictionary.MeanEnsembled object at 0x7f661c5aa740>
2023-09-24T13:01:13.383000+00:00 
2023-09-24T13:01:13.383000+00:00 The above exception was the direct cause of the following exception:
2023-09-24T13:01:13.383000+00:00 
2023-09-24T13:01:13.383000+00:00 Traceback (most recent call last):
2023-09-24T13:01:13.383000+00:00   File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
2023-09-24T13:01:13.383000+00:00     return _run_code(code, main_globals, None,
2023-09-24T13:01:13.383000+00:00   File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
2023-09-24T13:01:13.383000+00:00     exec(code, run_globals)
2023-09-24T13:01:13.383000+00:00   File "/opt/algorithm/process.py", line 102, in <module>
2023-09-24T13:01:13.384000+00:00     Unet_baseline().process()
2023-09-24T13:01:13.384000+00:00   File "/opt/algorithm/process.py", line 88, in process
2023-09-24T13:01:13.384000+00:00     monai_unet.run_inference(
2023-09-24T13:01:13.384000+00:00   File "/opt/algorithm/monai_unet.py", line 176, in run_inference
2023-09-24T13:01:13.384000+00:00     segment_PETCT(
2023-09-24T13:01:13.384000+00:00   File "/opt/algorithm/monai_unet.py", line 157, in segment_PETCT
2023-09-24T13:01:13.384000+00:00     ensemble_evaluate(post_transforms, models, device, val_dataloader)
2023-09-24T13:01:13.384000+00:00   File "/opt/algorithm/monai_unet.py", line 130, in ensemble_evaluate
2023-09-24T13:01:13.384000+00:00     evaluator.run()
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/evaluator.py", line 148, in run
2023-09-24T13:01:13.384000+00:00     super().run()
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/workflow.py", line 283, in run
2023-09-24T13:01:13.384000+00:00     super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 892, in run
2023-09-24T13:01:13.384000+00:00     return self._internal_run()
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 935, in _internal_run
2023-09-24T13:01:13.384000+00:00     return next(self._internal_run_generator)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 993, in _internal_run_as_gen
2023-09-24T13:01:13.384000+00:00     self._handle_exception(e)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 638, in _handle_exception
2023-09-24T13:01:13.384000+00:00     raise e
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 959, in _internal_run_as_gen
2023-09-24T13:01:13.384000+00:00     epoch_time_taken += yield from self._run_once_on_dataset_as_gen()
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 1087, in _run_once_on_dataset_as_gen
2023-09-24T13:01:13.384000+00:00     self._handle_exception(e)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 638, in _handle_exception
2023-09-24T13:01:13.384000+00:00     raise e
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 1068, in _run_once_on_dataset_as_gen
2023-09-24T13:01:13.384000+00:00     self.state.output = self._process_function(self, self.state.batch)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/evaluator.py", line 462, in _iteration
2023-09-24T13:01:13.384000+00:00     engine.fire_event(IterationEvents.MODEL_COMPLETED)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 449, in fire_event
2023-09-24T13:01:13.384000+00:00     return self._fire_event(event_name)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/ignite/engine/engine.py", line 425, in _fire_event
2023-09-24T13:01:13.384000+00:00     func(*first, *(event_args + others), **kwargs)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/workflow.py", line 226, in _run_postprocessing
2023-09-24T13:01:13.384000+00:00     engine.state.batch[i], engine.state.output[i] = engine_apply_transform(b, o, posttrans)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/engines/utils.py", line 261, in engine_apply_transform
2023-09-24T13:01:13.384000+00:00     transformed_data = apply_transform(transform, data)
2023-09-24T13:01:13.384000+00:00   File "/home/algorithm/.local/lib/python3.10/site-packages/monai/transforms/transform.py", line 171, in apply_transform
2023-09-24T13:01:13.384000+00:00     raise RuntimeError(f"applying transform {transform}") from e
2023-09-24T13:01:13.384000+00:00 RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f661c5ab280>