Container failure on main testing phases

Hi everyone,

We have seen multiple team's algorithm containers fail on main testing phase (primarily for Category 2 but some on Category 1 as well), even though they passed on preliminary phases. Upon looking into it, we have seen that all such containers are working fine on all videos except 1 which is a longer video as compared to others and the submissions errors out due to exceeding time limit set by grand-challenge for each run. We have reached out to GC support to see if this limit can be expanded and the containers can be re-run. In case that is not possible, we will remove that video from the test set and have all the teams (including those with successfull runs) re-run their containers which should hopefully work then. We will modify the dates for the deadline accordingly so that the teams can make their final submission. However, we still ask that the teams turn in their submission report on the set deadline.

Best, SurgToolLoc 2022 Organizing Committee

Dear organizer,

thanks for you reply.

After this problem is solved, will the container automatically re-run or we need to resubmit the container? This is very important because we have overlapped the original algorithm with a simpler but faster algorithm.

Best

Hi aneeqzia_isi,

Could you please check if our most recent failed submission to phase 2 was also due to this long video?

Team: NVIDIA Submitted by: boliu Submit time: Sept. 6, 2022, 10 p.m. New York time Error message: The algorithm failed on one or more cases.

I don't know the submission ID since there's no result link for failed submissions.

Thanks, Bo

Dear organizer,

We have the same problem. The containers, which passed on preliminary phases test fails on testing phase. Could you please check our submissions on testing phase?

Thanks, Ryo Fujii

Hi everyone,

Going to put up a separate forum post right after this, but the time limit for each video inference has been increased to 1 hr from 20 mins on the platform so hopfully this should be more than enough for the team's containers.

@boliu: yes, your most recent container failed due to time limit exceeded error as well. Please re-run to confirm if your container now runs successfully.

Best, SurgToolLoc 2022 Organizing Committee

Thanks aneeqzia_isi

I just re-submitted the container. It failed again showing the same error message: The algorithm failed on one or more cases.

Submit time: Sept. 7, 2022, 11:34 a.m. New York time

Can you take a look? Thanks!

Hi Boliu,

Your submission is still running out of time - can you confirm if you are using GPU for your inference?

Best, SurgToolLoc 2022 Org Committee

Can the hosts please tell us how many frames are in the long video so we can do some tests locally?

The longest video in the test set is ~64 mins long (or ~3840 frames). The limit set by GC per video is 1 hr so that means that the algorithm containers need to just slightly faster than 1fps at inference.

Best, SurgToolLoc 2022 Organizing Committee