Single and MultiThreadedAugmenter attribute error while training Unet

Hello I'm trying to train the UNet baseline. I followed all the steps in folder structure and data preparation. However, when running the train script (src/picai_baseline/unet/train.py) it shows an error for the MultiThreadedAugmenter:

This is the general configuration I'm using:

GPUs: NVIDIA Tesla P100-SXM2-16GB CPU Threads: 3 Dataset Definition: -------------------------------------------------------------------------------- Fold Number: 0 Data Classes: [0.0, 1.0] Train-Time Class Weights: [0.16634429 0.83365571] Training Samples [-:862;+:172]: 1034 Validation Samples [-:213;+:47]: 260 DataLoader - Image Shape: (8, 3, 20, 256, 256) DataLoader - Label Shape: (8, 1, 20, 256, 256) ---------------------------------------------------------------------------------------------------- * Loaded Neural Network Arch.: unet Start Training: Epoch 1

This is the error it shows for any number of max threads I use (except 1): Traceback (most recent call last): File "picai_baseline/src/picai_baseline/unet/train.py", line 162, in main() File "picai_baseline/src/picai_baseline/unet/train.py", line 135, in main model, optimizer, train_gen, tracking_metrics, writer = optimize_model( File "/workspace/picai/picai_baseline/src/picai_baseline/unet/training_setup/callbacks.py", line 107, in optimize_model for batch_data in train_gen: File "/workspace/picai/picai_baseline/src/picai_baseline/unet/training_setup/augmentations/multi_threaded_augmenter.py", line 200, in next item = self.__get_next_item() File "/workspace/picai/picai_baseline/src/picai_baseline/unet/training_setup/augmentations/multi_threaded_augmenter.py", line 188, in __get_next_item if not self.pin_memory_queue.empty(): AttributeError: 'MultiThreadedAugmenter' object has no attribute 'pin_memory_queue'

And if I select number of threads to be 1, another similar error appears on the SingleThreadedAugmenter: Traceback (most recent call last): File "picai_baseline/src/picai_baseline/unet/train.py", line 162, in main() File "picai_baseline/src/picai_baseline/unet/train.py", line 110, in main train_gen.restart() AttributeError: 'SingleThreadedAugmenter' object has no attribute 'restart'

Could you help me to understand what is happening? Thank you :)

Hi Manuel,

Your issue appears to be the same as discussed here. Did you try the proposed solutions there?

Kind regards, Joeran

Thank you for sharing that post, I did not notice that this problem was already commented. In fact, increasing the shared memory of the docker container to 16gb solved the problem. Thank you for your help and fast response.