some question about baseline

I downloaded baseline on Linux and selected one of network to run. As a result, its loss has a Nan value. Does anyone have this problem like me？ At present, I'm still solving the problem. Setting the learning rate to 0 can't solve this problem

Hey,

There are a couple things you need to consider why loss could go NaN in the baseline.

• Learning rate is too high. Try using 5e-5.
• Network loss is fluctuating quite a bit.
• Try using gradient clipping.

Hope that helps.

My learning rate has been set to 0, but this problem still exists, so it should not be the problem of too high learning rate I have found that in the train. py file, the Nan value will appear in the images of the inputs in some batches after pretreatment. If you encounter the same problem as me, you can refer to it

I guess some images may not be well saved when you run the preprocessing.

We had the same problem, solved by changing the version of Pytorch. In particular, we had Nan values when we tried Pytorch 1.12.1 (Stable). With version 1.8.2 (LTS), however, the training was successful. Actually, even in the prediction phase we got different results using different versions of Pytorch. Therefore, I suggest you check if you have the same problem by changing the version. I hope I have been helpful.

Thanks for sharing:)

Hi, I had the same problem. Unfortunately, changing the Pytorch version didn't work for me, but thank you for sharing. I got rid of the nan values by removing RandHistogramShiftd in the data augmentation part. Maybe it helps when someone else encounters the issue.

I got this message during pre-processing: " OME series is BinaryOnly, not an OME-TIFF master file." The baseline code with the default setting becomes Nan after ~20 epochs. I don't know if it is the reason.

I have met the same issue that the loss became Nan when training. I try one quick fix it to set squared_pred = False in DiceCELoss. Another fix would be to pass a larger value such as smooth_dr = 1e-5 to prevent division by zero. But both the two fixs are invalid.

@lmais

Hi, your method works for me. Thank you.

If you use monai version 0.9.0, it should also work with RandHistogramShift