The algorithm failed on one or more cases.

Hi organizers, I recently submitted my algorithm to the Grand Challenge platform, and while it works correctly during local testing, the platform returned the following error: "The algorithm failed on one or more cases."

Could you please provide me with the error log or any additional details regarding the failure?

Thanks for your assistance.

Hi,

Thank you for your submission.

Your algorithm encountered the following error:

Time limit exceeded

Please note that your algorithm has a five-minute limit to produce the segmentation of a single object, which should be sufficient time.

Additionally, we have the following suggestions for you:

1. First, run your inference.py file, located here. Then, prepare your Docker image for submission by following the guidelines provided here. Please note that you don't need to create the input and output folders when preparing the Docker image for submission. Also, do not use the inference file you use for local testing. Instead, use the inference file provided here.

2. If you encounter GPU memory issues when using sliding window inference, try transferring the data to the CPU instead of the GPU. You can do this as shown below:

   sliding_window_inference(image, (patch_size, patch_size, patch_size), num_samples, model, device='cpu')

I hope this helps resolve the issue.

Hello, could you please tell me how to submit the trained nnunet model? I have tried many times but was unable to submit it successfully.

Hello,

The issue is due to the size of the Docker container. Large Docker containers can take over 5 minutes to load, which exceeds the platform's time limit. This results in a timeout error with no logs displayed, only a timeout message.

From the information I obtained from Grand Challenge Support, the size of my previous containers (20-25GB) was far too large, and the images couldn't be pulled within the time limit. Most people are able to provide containers of under 10GB.

I resolved this by pulling a smaller image and setting up the environment with a final Docker size under 10GB.

I hope this helps.

Your advice has been very helpful, and I've made modifications accordingly. However, I've encountered some issues I'd like to consult you about. I would greatly appreciate it if you could reply.

Which Docker image did you use for inference? We initially used the devel image, but following your advice, we switched to runtime. Despite this change, we're still encountering errors.

How much time did the inference take locally for you? Using a GPU similar in performance to the T4, my local inference time is within 3 minutes, but I'm still encountering errors.

I also trained using nnunet. Would it be possible for you to provide an inference.py file as a template? I understand this request might be a bit much, so if it's inconvenient, it's okay.

Thank you very much for your assistance.

Thank you for your message. I'd be happy to assist with the issues you've encountered.

1. Docker Image for Inference: I used the initial Docker image ghcr.io/pytorch/pytorch-nightly:2.3.0.dev20240305-cuda12.1-cudnn8-runtime.

2. Local Inference Time: For local testing, I used an RTX 3090 GPU with 24GB of VRAM. The inference time was approximately 40 seconds per case. It's worth noting that my inference time results were obtained with use_mirroring=False. Enabling test-time augmentation (TTA) can significantly increase the inference time.

3. Softmax Code Modification: nnUNet's inference phase requires modifications to the softmax code to handle large tensors efficiently. I submitted a pull request detailing these changes, which you can review here: nnUNet Pull Request #2356.

4. Inference Script Template: You can refer to my inference.py file for guidance on implementing these modifications. The file is available at this link: inference.py.

I hope this information helps resolve the issues you're facing. I recommend trying out the algorithms and checking the logs before submission. However, although my submission was successful, the Dice scores differed significantly between the try-out phase and the submission phase. The results from the try-out phase matched my local testing, and I am still investigating the cause.

Thank you very much for your response. I will make the modifications according to your suggestions. I wish you achieve a good ranking.

Thank you very much for your generous help. I successfully submitted my solution through the docker you provided and received a reasonable score. Here are the issues I encountered, which I hope can help you:

• I indeed used nnUNet for training, but I was unable to successfully submit due to issues with using the docker during inference. Therefore, I ultimately used MONAI's sliding_window_inference for inference. In my offline scores, the inference results using this MONAI function were approximately 0.03 dice lower than the nnUNet inference framework.

• Since I could not see my inference results in the official test.sh, I previously ran the inference.py file directly in the local environment and compared it with the original images and labels using some visualization software. The results matched perfectly. Therefore, you can check whether the specific categories are predicted correctly and whether the positions correspond. (Although I believe you should have done this already).