The algorithm failed on one or more cases.

The algorithm failed on one or more cases.  

  By: syz on July 10, 2023, 3:53 a.m.

Dear organizers,

Thanks for hosting this challenge.

I encountered the problem "The algorithm failed on one or more cases." when my submit was going processing.

The test has passed offline. And I cannot get any error information of this problem.

Could you help me to solve the problem?

Best regards, syz

Re: The algorithm failed on one or more cases.  

  By: gasperpodobnik on July 10, 2023, 7:21 a.m.

Hi,

sure, the error messages are hidden from the participants to prevent potential information leaks about the test dataset. (Therefore it is the right thing to contact us in case of any errors in the evaluation process 🙂) The error message in your case is: numpy.core._exceptions.MemoryError: Unable to allocate 27.1 GiB for an array with shape (31, 204, 1072, 1072) and data type float32

Let me know if you need any further assistance in resolving this issue (you can also contact me via e-mail: gasper.podobnik@fe.uni-lj.si).

Best, Gasper

Re: The algorithm failed on one or more cases.  

  By: syz on July 10, 2023, 2:47 p.m.

Hi Gasper,

I have resubmitted a new algorithm, but unfortunately, I encountered the same error again. Could you please confirm if it is the same error as before, related to the memory allocation issue?

numpy.core._exceptions.MemoryError: Unable to allocate 27.1 GiB for an array with shape (31, 204, 1072, 1072) and data type float32

Your assistance in resolving this issue is greatly appreciated. If you need any further information or if there are any other errors, please let me know.

Thank you and best regards, syz

Re: The algorithm failed on one or more cases.  

  By: gasperpodobnik on July 11, 2023, 7:48 a.m.

Hi,

yes, it is a similar issue, this time you have reached GPU memory limitation: RuntimeError: CUDA out of memory. Tried to allocate 896.00 MiB (GPU 0; 14.76 GiB total capacity; 13.66 GiB already allocated; 223.75 MiB free; 13.68 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

You can try to increase the required GB memory (you can update this number in the algorithm settings or upload a new container and set a higher number, https://grand-challenge.org/documentation/making-a-challenge-submission/).

Best, Gasper

Re: The algorithm failed on one or more cases.  

  By: syz on Sept. 24, 2023, 3:37 p.m.

Dear organizers,

I encountered the problem "The algorithm failed on one or more cases." twice during those 2 days , when my submit was going processing.

The test has passed offline. And I cannot get any error information of this problem.

Could you help me to solve the problem?

Best regards, syz

Re: The algorithm failed on one or more cases.  

  By: gasperpodobnik on Sept. 25, 2023, 8:32 a.m.

Dear syz,

it looks like your submission ran out of memory. I'll check if we can increase the limits and will get back to you ASAP (I'll respond in the other forum topic that you opened).

Best, Gasper