Submission deadline approaching soon!

Hi teams,

The submission deadline for the preliminary testing phase is approaching soon (26th Aug). Please plan to start testing your algorithm submissions soon since teams are allowed 10 trials in the preliminary testing phase. We would like to encourage teams to make atleast one trial submission to the prelim testing phase by 19th Aug. There can be issues in submissions so having a working algorithm container soon will be critical in order to have a complete working submission. The final testing phase (in which the algorithm container will run on the main test set) will only have 1 trial submission per team.

Best, SurgToolLoc 2022 Organizing Committee

Dear organizer,

Hi, we tried to test our algorithm of Category 1 by uploading the container. We followed the instruction at the github repository, and the local testing worked fine. However, when we upload the container and submit it, The algorithm failed on one or more cases. is shown and the test failed. How do we debug our submitted container?

Best regards, Ryo

I tried to review the logs from the run on your algorithm container but I couldn't tell where it crashed since logs were empty. Could you add logging or print functions to your code so we can help debug where the code is breaking?

Hi kbot,

Our submission also failed with the message The algorithm failed on one or more cases. It was suscceful when we tested locally in docker.

We did have some print functions in the code. Could you share the print output with us privately so we could debug?

Thanks, Bo

Hi kbot,

Thank you for your help! I used try-out for Grand Challenge system and the stderr showed the following message, and it seems that all of the frames in the video are successfully processed. Could not find an int in the string 'endoscopic-robotic-surgery-video'.

The error is occured at this code. https://github.com/comic/evalutils/blob/main/evalutils/io.py#L45

It seems that the input video name should contain the numbers, but loaded video file "/input/endoscopic-robotic-surgery-video.mp4" does not have any number. I'm new to evalutils so that I'm not sure how to fix this error.

Best regards, Ryo

Hi Ryo,

Tryout will fail for our example container due to some issue with how Grand Challenge platform implements the Try-out in their backend. I'm still not sure about why actually and we still in the process of figuring out.

Regardless, the container should still work if you do a Prelim submission. The fact that Try-out fails for our example container is something we realized recently and have updated the Github page since (https://github.com/aneeqzia-isi/surgtoolloc2022-category-1). Like I mention previously, the container should still work for the prelim submission.

Due to this complication, we strongly recommend that you attempt at least one prelim submission with your actual algorithm during the Prelim phase so we can help debug issues that may arise.

Best, Challenge organizers

Hi kbot!

Thank you for your answer! We already upload our algorithm to Prelim Category 1 and the evaluation still fails... We have printed out some of the phases in our algorithm. Could you help debugging the algorithm?

Best regards, Ryo

Hi Ryo,

I've asked for permission to see your algorithm results so I can help troubleshoot.

Best, Kiran

Hi kbot,

thank you for your help. Let me know if you can see the log.

Best,

Ryo

Dear Organizing Committee,

What's the time limit of the docker run for final test phase? Can you provide some information on the test sets in terms of how many videos, how many total frames, and the type of GPU? This way we can budget total runtime to prevent time out in final test phase.

In case the only submission timed out in final test phase, do we get another submission until it's succesful?

Thanks, Bo

There are 85 videos in the test set of varying lengths. We will ask the Grand Challenge platform about timeouts and post here.

In reference to your question about allowing for further submissions due to technical challenges, this will be handled on a case-by-case basis as needed. I would recommend adding plenty of logging to the algorithm container in the situation that a case needs to be made that unexpected technical challenges should disqualify an entry.

Best, The organizers

Hi kbot,

Thanks for this information.

May I also ask the number of videos in Prelim stage Category 1?

aneeqzia-isi's sample submission which only predicts grasping_retractor and vessel_sealer for all frames scored 0.6292 in Prelim stage. This seems too high. There are 14 classes. 12 classes should have 0 score. Simple average f1 should score at most 2/14 = 0.1429. Did I miss anything?

Best, Bo

Hi boliu,

Do not rely on the performance on the preliminary test set to suggest performance on the full test set. The preliminary test set just has 2 videos and the full test set has 85.

Best, The organizers

@ryo-hachiuma and @zqlee0513

I see you had encountered the error: Could not find an int in the string 'endoscopic-robotic-surgery-video' in Try-out algorithm. I am currently experience the same issue. Can you share any advise how you resolved it?

@kbot, I've tried to make the submission with the algorithm as you suggested in prelim testing but it fails saying The algorithm fails one or more cases. Would please share the log of the container 8fa48926-6b7b-47cf-bcde-6fe741b40768 with me to debug this issue?

Looking forward to hearing fro anyone of you.

Regards,

@bilalUWE

022-08-25T14:11:13.292000+00:00 ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). 2022-08-25T14:11:13.293000+00:00 ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm). 2022-08-25T14:12:11.307000+00:00 Traceback (most recent call last): 2022-08-25T14:12:11.307000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1011, in try_get_data 2022-08-25T14:12:11.307000+00:00 data = self._data_queue.get(timeout=timeout) 2022-08-25T14:12:11.307000+00:00 File "/opt/conda/lib/python3.7/multiprocessing/queues.py", line 104, in get 2022-08-25T14:12:11.307000+00:00 if not self._poll(timeout): 2022-08-25T14:12:11.307000+00:00 File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 257, in poll 2022-08-25T14:12:11.307000+00:00 return self._poll(timeout) 2022-08-25T14:12:11.307000+00:00 File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 414, in _poll 2022-08-25T14:12:11.307000+00:00 r = wait([self], timeout) 2022-08-25T14:12:11.307000+00:00 File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 921, in wait 2022-08-25T14:12:11.307000+00:00 ready = selector.select(timeout) 2022-08-25T14:12:11.307000+00:00 File "/opt/conda/lib/python3.7/selectors.py", line 415, in select 2022-08-25T14:12:11.307000+00:00 fd_event_list = self._selector.poll(timeout) 2022-08-25T14:12:11.307000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler 2022-08-25T14:12:11.307000+00:00 _error_if_any_worker_fails() 2022-08-25T14:12:11.307000+00:00 RuntimeError: DataLoader worker (pid 85) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit. 2022-08-25T14:12:11.307000+00:00 2022-08-25T14:12:11.307000+00:00 The above exception was the direct cause of the following exception: 2022-08-25T14:12:11.307000+00:00 2022-08-25T14:12:11.307000+00:00 Traceback (most recent call last): 2022-08-25T14:12:11.307000+00:00 File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main 2022-08-25T14:12:11.307000+00:00 "__main__", mod_spec) 2022-08-25T14:12:11.307000+00:00 File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code 2022-08-25T14:12:11.307000+00:00 exec(code, run_globals) 2022-08-25T14:12:11.307000+00:00 File "/opt/algorithm/process.py", line 209, in 2022-08-25T14:12:11.307000+00:00 Surgtoolloc_det().process() 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/evalutils/evalutils.py", line 183, in process 2022-08-25T14:12:11.308000+00:00 self.process_cases() 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/evalutils/evalutils.py", line 191, in process_cases 2022-08-25T14:12:11.308000+00:00 self._case_results.append(self.process_case(idx=idx, case=case)) 2022-08-25T14:12:11.308000+00:00 File "/opt/algorithm/process.py", line 123, in process_case 2022-08-25T14:12:11.308000+00:00 scored_candidates = self.predict(case.path) #video file > load evalutils.py 2022-08-25T14:12:11.308000+00:00 File "/opt/algorithm/process.py", line 165, in predict 2022-08-25T14:12:11.308000+00:00 tta_res.append(learn.tta(dl=learn.dls.test_dl(fs))) 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/fastai/learner.py", line 645, in tta 2022-08-25T14:12:11.308000+00:00 aug_preds.append(self.get_preds(dl=dl, inner=True)[0][None]) 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/fastai/learner.py", line 290, in get_preds 2022-08-25T14:12:11.308000+00:00 self._do_epoch_validate(dl=dl) 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/fastai/learner.py", line 236, in _do_epoch_validate 2022-08-25T14:12:11.308000+00:00 with torch.no_grad(): self._with_events(self.all_batches, 'validate', CancelValidException) 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/fastai/learner.py", line 193, in _with_events 2022-08-25T14:12:11.308000+00:00 try: self(f'before'); f() 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/fastai/learner.py", line 199, in all_batches 2022-08-25T14:12:11.308000+00:00 for o in enumerate(self.dl): self.one_batch(*o) 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/fastai/data/load.py", line 129, in iter 2022-08-25T14:12:11.308000+00:00 for b in _loadersself.fake_l.num_workers==0: 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 530, in next 2022-08-25T14:12:11.308000+00:00 data = self._next_data() 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1207, in _next_data 2022-08-25T14:12:11.308000+00:00 idx, data = self._get_data() 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1173, in _get_data 2022-08-25T14:12:11.308000+00:00 success, data = self._try_get_data() 2022-08-25T14:12:11.308000+00:00 File "/home/algorithm/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1024, in _try_get_data 2022-08-25T14:12:11.308000+00:00 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e 2022-08-25T14:12:11.309000+00:00 RuntimeError: DataLoader worker (pid(s) 85) exited unexpectedly