Submission failed - next submission within current 7 days

Hii,

our submission failed as well, and we would also be interested in a log msg, why our algorithm could not be sumbmitted? (The algorithm failed on one or more cases).

Also, another question: does the failed submission count as our submission for the next 7 days? And would it be possible to apply this limitation only to successful submissions?

Thanks, Miriam

Hi Miriam,

Your submission failed because it attempted to access the internet. Please note that your Docker container will not have access to the internet when executing on grand-challenge.org. See the Qualification submission tutorial for more information.

I am uncertain whether failed submissions count. If you are unable to make another submission right now, please let us know.

Best, Luuk

Hi Luuk,

thank you for ur information! Alright seems like we missed this detail. Perfect, I'll let you know if we encounter any more submission problems within the next 7 days.

Best, Miriam

Hi Luuk,

i believe we can close the issue, we've just uploaded a new active container image, and it's queued for the leaderboard :D

Best, Miriam

Hi Luuk,

our submission failed again, could you tell us why?

Thanks, Miriam

Hi Miriam, your container ran flawlessly on most CT scans, but it failed with the error: The container was killed as it exceeded the memory limit of 30g.

Hi Luuk,

thanks for the information, we'll look into it.

Best, Miriam

Hi Luuk,

we've tried to solve the memory issue, but the algorithm failed again, could you tell us why?

Merci, Miriam

Hey @LuukBoulogne,

first of all, thank you for your time and effort in helping us to identify this issue.

We tried to reproduce this memory leak on mulitple machines without any success, sadly. Applying the container locally on the training set resulted into a RAM usage of < 16GB (~8-11GB) for utilizing our Tensorflow model.

As @miriamelia already said, would it be possible that you can give us some more information like exact error messages or after how many samples the error occurs?

What we currently tried: - We observed that Tensorflow is piling up some data after each predict() call - To solve this issue we are now running building the TF stack for each prediction in a separate process - This costs of course prediction time, but allows us to clean up all leftover variables in the RAM (Garbage collection on TF graphs or session clear is still not fixed by TF sadly) - Furthermore, we manually reduced the number of threads to 5 (instead of the default TF option to use all core threads it can get) in order to save more RAM and ensure the --pids-limit

Also thank you for the great organization of this challenge!

Cheers, Dominik

Hi Miriam and Dominik,

I am happy to help. The error of last time has gone away. It now failed because of a different error: "Time limit exceeded." Your Algorithm still processes most CT scans without issues. The processing of a single CT scan by a submitted Algorithm should preferably take no more than 5 minutes on an NVIDIA T4 GPU (16 GB) with 8 CPUs (32 GB). We implemented a hard limit of 20 minutes for a single image. A detail of the evaluation process that might be good for you to know is that your docker container will start up separately for each CT scan in the test set. This means that in the evaluation process, there will only be one predict call per run. Note that this means that the inference time includes loading your model.

Best, Luuk

Hi Luuk,

alright, thank you for the information! We'll continue searching for a solution :D

Best, Miriam