The validation dataset should not be used

How will you check if teams are incorporating validation sets into their training?In theory, the results should improve！

We believe that all participating researchers will abide by the rules to have a fair analysis of the challenge results. Still, any suspicious results will be requested for code submission for verification of results. All top-performing methods will also submit a report and present the result at the challenge event.

Thank you very much for your reply, but how will suspicious results be defined? I tried to join the verification set for offline training, and the results can be greatly improved! Therefore, if I choose to add part of the verification set, my ranking score can be controlled within a certain range at will. The main reason is that there is a problem in the division of competition data. The verification set and the test set adopt exactly the same data distribution. In the training stage, the verification set or the test set with the same data distribution should not be published. As a competition, we should ensure absolute fairness. I think you can either open the verification set training or replace an independent test set. The players will retest. We look forward to your reply! thank you!

I also agree that using validation data for training greatly improves performance. (Because validation and test data are assumed to be the same patient)

Therefore, the organizer also emphasized that validation data should not be used, So all participants are considered to participate in the competition by faithfully following the rules. (Of course, We strictly abides by this rule.)

If all or part of the validation data is used, the score will rise dramatically, and even if it does not rise dramatically, an abnormal rise will be detected in some classes. And if the organizer re-learns based on the codes of the winners after the competition, it will be possible to immediately determine whether the score is from the validation data or not. (Of course, it is difficult to achieve 100% reproducibility due to server, package version, etc., but if validation data is used, the score gap will widen beyond common sense)

We understand your concern about any participant using unfair means to gain the rank. Your concerns have been forwarded to organising team. We want to reassure your that all preventive measures will be taken to maintain fairness in the challenge. We will soon communicate further details on the validation of results and invitations to challenge event at MICCAI 2021.