Request for logs of failed submissions

Dear orgnization team,

I have tried three submissions with different containers for the preliminary test, but all submissions got the failed results with 'The algorithm failed on one or more cases'. My containers are able to get the correct log files on the try-out. Could you please show me the logs of my failed submissions for my debugging?

Their evaluation IDs are as follows: f9e94699-7752-437c-aed4-bd648e87defe 7ec0ffd7-0bb2-473d-8254-dfc66a75c882 23af34df-e074-477e-8563-38e23bb6bbaf

Thank you very much for your help in advance! Best

The evaluation logs all say the same thing

Logs Stdout No logs found on stdout Stderr No logs found on stderr

Did you take a look at the algorithm log? It is accessible to you from the algorithm page, under the Results category.

Could you share the logs here from the algorithm run?

Best, The organizers

Hi kbot,

Thank you for your reply.

The algorithm logs about stderr are as follows:

2022-08-23T13:46:43.547000+00:00 Could not find an int in the string 'endoscopic-robotic-surgery-video'.

I did not get stderr like this when test on my local environment. Do you have any ideas for solving the error?

Thank you! Best

Could you add logging to your algorithm container to see whether sub-processes are completing as expected? Did it say that it succeeded?

Best, The organizers

Hi,

I am getting the same error during testing out the algorithm:

2022-08-23T23:29:07.377000+00:00 Could not find an int in the string 'endoscopic-robotic-surgery-video'.

Hi kbot,

Thank you for your help.

According to the stdout, the process was successfully executed and the correct outputs were generated.

Thank you! Best

I was able to check your results for the failed runs. It seems it was a CUDA out of memory error.

RuntimeError: CUDA out of memory. Tried to allocate 1.58 GiB (GPU 0; 14.76 GiB total capacity; 11.72 GiB already allocated; 1.30 GiB free; 12.54 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

This is accessible to you actually and not me. I could only see after you granted me permission for the algorithm container.

You have to access the results of you algorithm container and then click through on where it says "Failed" in red to check the run logs. Is this what you did previously? Hopefully, I'm able to provide new information.

Best, The organizers

Hi kbot,

I really appreciate your help and the information about the CUDA out memory error.

I checked my algorithms interfaces and on the 'Results' page I can only access the results of the Try-out. As I said before, my Try-outs succeeded so I only get the stderr of 'endoscopic-robotic-surgery-video'. Unfortunately, on the 'Results' page, I do not find the results of my submissions for the preliminary test. I got some results on the 'Submissions' page and there are some red 'Failed' there. However, it is pure text and not linked to any run logs.

In addition, I have also submitted a test whose container only uses CPU without GPU. I still got the failed. So I think the error may not fully cuased by the CUDA out of memory. Can you offer the log about my latest submission?

Hoping you can offer further information about how to get the failed logs.

Thank you very much! Best

Hi zqlee0513,

I see you have been able to make a submission finally. How were you able to resolve the 'could not find an int in the string 'endoscopic-robotic-surgery-video' error?

Thanks, Tao

Hi @tdakinosho,

I only succeeded with submitting the original container, where on model is loaded and the outputs are fixed with "needle_driver"=True and "monopolar_curved_scissor"=True.

Best

Hi @zqlee0513,

The bug causing this error may have been fixed as stated in this thread: https://grand-challenge.org/forums/forum/endoscopic-surgical-tool-localization-using-tool-presence-labels-663/topic/bug-fix-prelim-category-2-evaluation-container-relating-to-keyerror-suction-irrigator-1198/

If you still want to have the logs, please send me: 1 - Algorithm name. 2 - Category name. 3 - ID of the submission attempt for the algorithm. 4 - Team name. 5 - The username who submitted the algorithm for evaluation.

I appreciate your patience!

Best, SurgToolLoc 2022 Organizing Committee