Open Leaderboard

Hi,

first of all i wanted to thank the organizers again for this great challenge and especially for providing a open leaderboard.

Many leaderboards in the field of medical imaging are overloaded with many submissions and it is not clear which method belongs to which submission. Very often people don't stick to the rules of the original challenge and use pretrained models, additional data, crazy big ensembles or overfit the test set with hundreds of submissions. Some of these approaches are justified, but in order to make a fair comparison of one's own methods, it would at least be important to know what one is comparing to.

To prevent this from happening, it would be helpful if some information were requested for each submission: - Single model or ensemble? - Pretrained model? - Additional data used? - Trained only on training data or as well on validation data? - Possibility to add a link to a publication (even later?) ....

Probably, the community could provide more important aspects. I would love to get your feedback :)

Best, Constantin

Dear Constantin,

Sorry for late reply, I've been busy submitting my paper and attending conferences over the past few weeks, so I haven't been able to check the information in time, and I'm very happy to agree with you that it's time for a truly fair comparison of leaderboards for real development in the field of medical analytics. Researchers would need to submit real information in the form of questionnaires, including network descriptions, training details, etc., and we would need human review before the results would appear on the leaderboard, again requiring self-honesty on the part of the submitter, but we would like to have more definable details to help with subsequent studies.

Thank you very much for your suggestion, and a formal ranking will be available soon!