Category 2 update: "probability" field

Hi everyone,

There's been an update to the Category 2 challenge submission format. We have added a new required field for the output json. Please include a "probability" entry to include the confidence score for each detected bounding box.

See the updated Submission Instructions page for further info.

Best,

SurgToolLoc 2022 Organizing Committee

Hi SurgToolLoc 2022 Organizing Committee,

We submitted to Category 2 successfully but got 0.0 mean_mAP score.

I saw that two other teams have the same issue. https://grand-challenge.org/forums/forum/endoscopic-surgical-tool-localization-using-tool-presence-labels-663/topic/problems-of-evaluation-results-1133/

Could you check whether it's due to our bbox format not being correct? The predictions on vid_1_short.mp4 look correct. It should have higher than 0 score.

Thanks, Bo

Dear Organizer,

We have the same problem. Would you check the evaluation system in Category 2 ?

Regards, Satoshi

Hi everyone,

There is an issue with the evaluation of Prelim Category 2 that is now known to the organizers. The organizers are currently working to address this and will send out an email to participants when this issue has been fixed.

The container is still fine to test whether input and output files are being processed correctly. However, the mean_mAP is being computed in a way that is incorrect. That is why we suspect everyone has a mean_mAP near zero.

Thank you everyone for bringing this to our attention!

Best, The organizers

Thanks kbot.

Do you expect Category 2 metric to be fixed before the end of Prelim stage? Will the Prelim stage be extended to allow teams time to test it after it's fixed?

Thanks, Bo

The evaluation metric in prelim category 2 is being computed incorrectly. Otherwise, prelim category 2 submission should be a good test to ensure that the correct output files are generated by your submission. We hope to resolve the incorrect calculation of the output metric soon.

Currently, you should still be able to make a submission to Prelim Category 2 and generate a result. However, the result will not be correct right now.

We have already increased the total number of submissions available to participants for Prelim phase from 10 to 15. We may increase the the end-date of the phase Prelim phase based on how quickly we can release the fix.

Best, The organizers