Timeline, validation set and submission

Hi,

I have a few quick questions:

1. Is the testing set that will be relased (without labels) the same that will be used for the final ranking? Or is this a "validation" set and there is a separate secret test set for the final ranking?
2. Once the submissions open, will we get to see our performance on the test set already? If it's the final test set, then of course we shouldn't, but it might be useful to have a separate validation set just so we can check that the performance makes sense and nothing has gone wrong with the data. This validation set could even be a copy of the training data for this purpose.
3. When inspecting the training data manually, there was something I found curious. Is it okay to ask questions about the data here or would the organisers prefer that we don't before the deadline?

Thanks for organising this challenge!

Best wishes, Justin

Hi, Thanks for your valuable questions. 1. The testing set that will be relased is the same that will be used for the final ranking. 2. You can see your performance on the test set once you submit your results. We will not provide the validation set, you can split some data from training data as the validation set. We will provide the evaluation metrics used in this challenge, and you can use it to check your results. 3. For the questions of the data, you can email us through our email: drac22miccai@163.com, we will happy to discuss it.

Best wishes

Would be nice if all questions will be publicly discussed and not in private between teams and organizers to ensure a fair competition :)

Ok, I agree with you. You can discuss these questions publicly.

I also agree and I'm happy to share questions and knowledge - if others want to do so too, that'd be great. But I wanted to check that it's okay with the organisers.

I've noticed that images 106 and 107 seem to be of the same eye. There are minor differences and the DR label has also changed between the images. Are these two images of the same person at different points in time and has the disease indeed progressed between visits?

Cheers!

Hi,

We have noticed this problem. After confirming with the doctor, images 106 and 107 are both grade 0 in task 3 (DR Grading). You can correct that by yourself, and we will inform other participants.

Thank you.

DRAC2022 Organizers