Unet Traning- Error

Unet Traning- Error  

  By: marif257 on Aug. 4, 2022, 12:52 p.m.

Hi, It maybe a basic question but need your help.

I am getting following error when I run train.py. usage: train.py [-h] [--max_threads MAX_THREADS] [--validate_n_epochs VALIDATE_N_EPOCHS] [--validate_min_epoch VALIDATE_MIN_EPOCH] [--export_best_model EXPORT_BEST_MODEL] [--resume_training RESUME_TRAINING] --weights_dir WEIGHTS_DIR --overviews_dir OVERVIEWS_DIR --folds FOLDS [FOLDS ...] [--image_shape IMAGE_SHAPE [IMAGE_SHAPE ...]] [--num_channels NUM_CHANNELS] [--num_classes NUM_CLASSES] [--num_epochs NUM_EPOCHS] [--base_lr BASE_LR] [--focal_loss_gamma FOCAL_LOSS_GAMMA] [--enable_da ENABLE_DA] [--model_type MODEL_TYPE] [--model_strides MODEL_STRIDES] [--model_features MODEL_FEATURES] [--batch_size BATCH_SIZE] [--use_def_model_hp USE_DEF_MODEL_HP] train.py: error: the following arguments are required: --weights_dir, --overviews_dir, --folds

I have tried to pass the arguments but they way you have described does not work.

python -u src/picai_baseline/unet/train.py \ --weights_dir='/workdir/results/UNet/weights/' \ --overviews_dir='/workdir/results/UNet/overviews/' \ --folds 0 1 2 3 4 --max_threads 6 --enable_da 1 --num_epochs 250 \ --validate_n_epochs 1 --validate_min_epoch 0

Please note I have created an enviroment and am using pycharm. I am trying to run it in Editor directly.

if I also used anoconda prompt in the respective drive/directory and call the trian.py wit the respective arguments it is following error. 
    device, args = compute_spec_for_run(args=args)

File "c:\windows\system32\picai_baseline\src\picai_baseline\unet\training_setup\compute_spec.py", line 23, in compute_spec_for_run args.num_threads = np.min([len(os.sched_getaffinity(0)), args.max_threads]) AttributeError: module 'os' has no attribute 'sched_getaffinity' Could you please help me how to pass these arguments in the pychame if I directly run thre trian.py

Best regards
 Last edited by: marif257 on Aug. 15, 2023, 12:57 p.m., edited 1 time in total.

Re: Unet Traning- Error  

  By: anindo on Aug. 4, 2022, 9:18 p.m.

Hi Muhammad,

It seems that 'os.sched_getaffinity' isn't supported by all operating systems (e.g. Windows OS). I've updated picai_baseline, such that if 'os.sched_getaffinity' isn't supported, the script defaults to the value of the argument '--max_threads' in train.py instead. So please reinstall picai_baseline, and remember to set '--max_threads' such that it doesn't exceed the number of CPU cores/threads available in your setup.

Hope this helps.

 Last edited by: anindo on Aug. 15, 2023, 12:57 p.m., edited 1 time in total.

Re: Unet Traning- Error  

  By: marif257 on Aug. 11, 2022, 1:20 p.m.

Hi Anindo,

Thank you for updating the picai_baseline. It works fine now.

I have another question regarding labels. There are two types of labels provided. One with human expert and one with AI. My question is, are the binary labels provided by AI contains any false positive? I mean do the annotated labels of 220 csPCa patients by human experts match with the AI labels. Also the remaining 205 csPCa patient AI lables are true positive and can be used for training and reference ground truth.

I hope to hear you soon.

Re: Unet Traning- Error  

  By: marif257 on Aug. 11, 2022, 3:20 p.m.

Hi,

I ma getting error for 2 cases when I tried to processed only AI generated labels.

Processed 1498 items, with -2 cases skipped and 2 errors.

Did not generate dataset.json, as only 1498/1500 items are converted successfully.

One case is 11114_1001137.nii.gz, while otherone I dont know.

Could you please have a look and let me know.

Re: Unet Training- Error  

  By: joeran.bosma on Aug. 12, 2022, 10:14 a.m.

Hi Muhammad,

In short, the AI-derived annotations (found in picai_labels/csPCa_lesion_delineations/AI/Bosma22a/) contain as many lesion delineations, as the number of ISUP ≥ 2 lesions reported for that case in its associated pathology report. Whether or not these lesion delineations will match the location and spatial extent of the reported ISUP ≥ 2 lesions is something we cannot guarantee, and remains a limitation of using these automatic annotations.

Please refer to (Bosma et. al., 2022) for more details on the methodology that we used to create these annotations. However, do note one distinction: in the paper, we use the number of PI-RADS 4-5 lesions reported in the radiology report of a given case; for the PI-CAI training datasets, we use the number of ISUP ≥ 2 lesions reported in the pathology report. Additionally, we used the nnU-Net baseline model to generate lesion candidates.

So, to answer your question: no, the AI-derived lesion annotations do not match the human expert annotations completely. You can compare the two types of annotations using picai_eval. For the 5-fold cross-validation folds with human expert annotations, the performance of the automatic annotations can be found here.

As for your preprocessing error, I would need to see the logfile to identify the error. It may be because of the preprocessing error discussed here. If so, please refer to the proposed solutions.

Hope this helps, Joeran

Docker container error  

  By: marif257 on Sept. 29, 2022, 2:55 p.m.

Hi, Tried to train nnUnet using docker container but I am getting following error. Any suggestions.

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: initialization error: WSL environment detected but no adapters were found: unknown.

Re: Unet Traning- Error  

  By: joeran.bosma on Sept. 29, 2022, 4:15 p.m.

Hi Muhammad,

It appears you have an issue with your Docker installation, rather than an issue specific to the Docker containers provided in the PI-CAI Challenge. Are you able to run the hello-world example container from the Docker installation guide?

Kind regards, Joeran

Re: Unet Traning- Error  

  By: marif257 on Sept. 30, 2022, 9:58 a.m.

Hi Joeran,

Yes, I can run and see below.

docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 2db29710123e: Pull complete Digest: sha256:62af9efd515a25f84961b70f973a798d2eca956b1b2b026d0a4a63a3b0b6a3f2 Status: Downloaded newer image for hello-world:latest

Hello from Docker! This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

Also I can see this container is in my docker dekstop terminal. However, when I pull or build the docker container as you said on webpage the following message appear but it does not show in the docker deskstop terminal.

docker pull joeranbosma/picai_nnunet:latest latest: Pulling from joeranbosma/picai_nnunet Digest: sha256:853e828aa57a32f38ef17fe1ba6adb86665a3f19d07f929265dea1d5cd0b2227 Status: Downloaded newer image for joeranbosma/picai_nnunet:latest docker.io/joeranbosma/picai_nnunet:latest

Following message when I build the container.

docker build . --tag joeranbosma/picai_nnunet:latest [+] Building 4.7s (21/21) FINISHED => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 38B 0.0s => [internal] load .dockerignore 0.1s => => transferring context: 35B 0.0s => [internal] load metadata for nvcr.io/nvidia/pytorch:20.12-py3 3.5s => [internal] load build context 0.1s => => transferring context: 224B 0.1s => [ 1/16] FROM nvcr.io/nvidia/pytorch:20.12-py3@sha256:cc14c0cf580989bb1ff39fa78ca697b77a8860b17acead4a60b853bb45499f8d 0.0s => CACHED [ 2/16] RUN groupadd -r user && useradd -m --no-log-init -r -g user user 0.0s => CACHED [ 3/16] RUN echo "PYTHONUNBUFFERED=1" >> /etc/environment && echo "OMP_NUM_THREADS=1" >> /etc/environment 0.0s => CACHED [ 4/16] RUN apt-get update && apt-get install -y --no-install-recommends graphviz && apt-get clean && rm -rf /var/lib/a 0.0s => CACHED [ 5/16] RUN git config --global advice.detachedHead false && git clone https://github.com/DIAGNijmegen/nnUNet.git /home/user/nn 0.0s => CACHED [ 6/16] COPY io.py /tmp/io.py 0.0s => CACHED [ 7/16] RUN SITE_PKG=pip3 show nnunet | grep "Location:" | awk '{print $2}' && mv /tmp/io.py "$SITE_PKG/nnunet/utilities/io.p 0.0s => CACHED [ 8/16] RUN pip3 install picai_eval picai_prep git+https://github.com/DIAGNijmegen/picai_baseline 0.0s => CACHED [ 9/16] COPY nnUNetTrainerV2_focalLoss.py /tmp/nnUNetTrainerV2_focalLoss.py 0.0s => CACHED [10/16] RUN SITE_PKG=pip3 show nnunet | grep "Location:" | awk '{print $2}' && mv /tmp/nnUNetTrainerV2_focalLoss.py "$SITE_PK 0.0s => CACHED [11/16] COPY nnUNetTrainerV2_Loss_CE_checkpoints.py /tmp/nnUNetTrainerV2_Loss_CE_checkpoints.py 0.0s => CACHED [12/16] RUN SITE_PKG=pip3 show nnunet | grep "Location:" | awk '{print $2}' && mv /tmp/nnUNetTrainerV2_Loss_CE_checkpoints.py 0.0s => CACHED [13/16] COPY nnUNetTrainerV2_Loss_FL_and_CE.py /tmp/nnUNetTrainerV2_Loss_FL_and_CE.py 0.0s => CACHED [14/16] RUN SITE_PKG=pip3 show nnunet | grep "Location:" | awk '{print $2}' && mv /tmp/nnUNetTrainerV2_Loss_FL_and_CE.py "$SI 0.0s => CACHED [15/16] COPY --chown=user nnunet_wrapper.py /home/user/nnunet_wrapper.py 0.0s => CACHED [16/16] RUN chmod +x /home/user/nnunet_wrapper.py && ln -s /home/user/nnunet_wrapper.py /usr/local/bin/nnunet 0.0s => exporting to image 0.7s => => exporting layers 0.0s => => writing image sha256:70ce2833155447f50ed9db46d2b39245430a44fecfd01f8885e8024406927771 0.0s => => naming to docker.io/joeranbosma/picai_nnunet:latest 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

Any help will be apprecaited. Thank you.

 Last edited by: marif257 on Aug. 15, 2023, 12:57 p.m., edited 2 times in total.

Re: Unet Traning- Error  

  By: joeran.bosma on Sept. 30, 2022, 12:34 p.m.

Hi Muhammad,

Thanks for verifying Docker works as expected.

When I pull/build the Docker container, it does show up in the Docker Desktop program, under "Images".

I am afraid I cannot help you further with this issue. There is something going wrong between WSL(2) and nvidia/GPU support/the specific Docker. I do not have a Windows machine, so I cannot debug this.

Kind regards, Joearn