Unfair？

Dear,

Why can people in a team submit multiple times a day from different accounts? For a person like me, it is obvious that others submit a lot more than us, which is obviously unfair to us. Even if it is a different plan, it should not be allowed. It is only fair that a team submits once a day. And you can't tell if someone else's approach is different.

Thanks Wang

This is a screenshot

Dear Wang,

submission for the same algorithmic approach is against the rules of the challenge, as it is a means of circumventing the limitation imposed by the challenge and can lead to disqualification from the challenge.

Further, it is true that there are many ways in which some participants will have an advantage over others, being part of a team is one of them. But having ample compute resources is another one. We could now discuss how we could equalize for each of them, but finally, I think there will be no complete fairness.

Ultimately, the preliminary test phase was 3 weeks long, and I would argue that having, e.g. 40 over 20 trials does not add a significant competitive advantage, given that the leaderboard order on the preliminary test set is ultimately not of importance and the final test set is completely different, in that no combination of tissue/tumor type and species will be recurring from the preliminary test set. So my personal expectation is that fine-tuning hyperparameters on the preliminary test set could very well backfire.

Best,

Marc

Dear Marc, 1. I can understand your point of view, but I have a few questions, how do you judge that every time the same team submits a similar proposal. 2. According to the experience of last year, this preliminary test list is positively correlated with the final test set, so the same team can be used to select a more reasonable model than we are alone. Of course, I participated in so many large-scale competitions. From our experience, in almost all competitions, such as kaggle or grand challenge, the same team shares the number of submissions, not the more people in the team can enjoy more times.

Thanks Wang

Dear Wang,

thanks for understanding. To answer your questions:

1. There are a few tricks for finding out and we're in close touch with grand-challenge on such issues. Last year, this led to a disqualification within our challenge.
2. You were right in that we had a high correlation between the two test sets last year. This was caused by using the same scanners for the preliminary test set and the final test set and was the reason why we designed the test sets differently this year.

We assume that the majority of people are fair contestants and obey the rules. After all, they are publishing a preprint with their full name and institution on it so they have quite an incentive to not do anything embarassing. On top of that: If you want to break the rules, it is much easier with a restriction on the team than on the user, since it is very easy to create, join and leave teams. So in fact, this would even allow for more frequent submissions.

Best,

Marc

Dear Marc,

Thank you very much for your reply, and thank you very much for hosting such a great competition, hope to have the opportunity to cooperate in the future.

Thanks Wang

Dear Marc, Thanks for clarifying the issue and explaining the situation.

Dear Wang, I want to ensure you that the two algorithms submitted by our team are different and related to different tracks of the challenge. Technically, we have submitted these two algorithms in two days (we are based in the UK, one algorithm was submitted on Monday and the other one on Tuesday but because time is not shown on the challenge website you cannot tell), although we were allowed to do it on the same day. As Marc explained, this is one of the advantages of participating as a team and we are not breaking any rules as long as the two algorithms are different. Furthermore, we have the same idea as Marc, that overfitting on this preliminary test set is not a good idea, at all. You see, the current preliminary test set is very small, from a different population in comparison to the final test, and contains fake plain images. I wouldn't fine-tune my algorithm based on this test set even if I were allowed to do so. And if you look at the leaderboard, we have only made 2 submissions so far and are not planning to do much more, to be honest. I understand participating in a challenge as an individual can be stressful and overwhelming. I hope these explanations reassure you and you are not feeling "unfair" anymore.

Best wishes, Mostafa

p.s. Leaderboard for me shows submission on different days:

Dear Mostafa, Thank you very much for your reply, my pressure is very low, I have won at least 10 championships in medical image competitions, including many pathology competitions, the RSNA2019 champion. I just like to enjoy the process of the competition more, and don’t care about the ranking. It’s just that I started It's weird, it's not like the big competitions I've been in before.

Thanks Wang