Finalized leaderboard will be announced on the day of challenge - Sept 18th

Hi everyone,

We are still in the process of finalizing the submission from all teams and still waiting on some teams to submit slides/recorded talks. Due to these delays, we will be announcing the finalized leaderboard on the day of challenge i.e Sept 18th, during the presentation and on the website.

Best, SurgToolLoc 2022 Organizing Committee

Hi, all your deadlines have been postponed.

So far it was because of problems on your side: wrong input spec, wrong metric definition, wrong metric computation, too slow evaluation code, etc. We all played by your constantly changing rules without complaining much. And now this.

What is the reason for this one?

If it is because some teams have not delivered their content in time then they should not qualify. If you accept them then it is unfair as some teams would have got more time than others. This is unfair to all teams who met your deadlines.

Besides potential unfairness, you require top teams to present at the workshop, yet you do not announce which teams are top teams before the workshop. Do you assume people will take the risk to come and not appear on the leaderboard? Can you at least publish now he top 3 or top 5 teams in each task?

Last, but not least, people on our team, and probably on other teams, assumed your deadlines were true deadlines and had other plans after these. Given you postponed all deadlines very lately, these plans were jeopardised. For instance some people had to cancel vacations with their families for instance.

I hope you will ensure fairness in team evaluations, and I also hope you will learn from the organisation issues we had to deal with.

Dear JFPuget,

Thank you for your valuable feedback.

Please be assured that the time given to all teams was THE SAME. The deadlines were extended because there were issues relating to hosting platform that were out of our hands and had to get GC support involved very frequently. As this was the first time a video based challenge was being hosted on GC, there were delays in getting everything set up initially.

We would recommend you to go back and see the signed form your team sent in order to access the data in which you will find the clause "To be present at the challenge with at least one team member and present your method." So even before each team was given data access and be willing to participate, regardless of winning status, there had to be presence of each team (even if it was one member virtually). So we hope that all teams took the time to look through the agreement and plan their work/vacations accordingly if they were participating.

Following up on the above point, we had to ensure all teams that had submitted algorithms submitted everything that was required and had to gather information relating to their registration status and presentation/recorded videos. As there were many teams that participated, there were delays in gathering this information which we did not anticipate. Hence we decided to announce the results on challenge day (which is very normal for such challenges). Again, please be assured that no team was allowed to submit their algorithm container after the deadline. We were flexible in few timelines to ensure fairness for all teams (including yours) since there were delays that were not in our hands.

We learned a lot from organizing this challenge and will definitely try to improve on issues that teams faced in future challenges and hopefully have a smoother challenge experince for everyone.

Best, SurgToolLoc 2022 Organizing Committee