ungradable | referability

Dear organizers, I would like to clarify the expectations of the output:

The concern is that if the image is ungradable ("poor quality / unusable"), it seems inappropriate to make a diagnosis (referable / unreferable) using this image.

Thus, in the case the image is ungradable (O3 = true), what shall be the "referability" return for O1 and O2 (i.e. null or false)?

Best Regards, Dmitrii

Dear Dmitrii,

If the label of an image is indeed ungradable, your produced outputs for O1 and O2 are ignored during evaluation. So in those cases it does not matter what you output there, as long as O1 is a float and O2 is a boolean, so not null or false. However, if your O3 is true, while the label for the image is actually "ungradable", O1 and O2 will be used for evaluation.

Best regards, Coen de Vente

Dear organizers: According to the evaluation code, I would like to ask if my explanation is correct

First. O1 is used to calculate the partial area under the receiver operator characteristic curve (90-100% specificity) for referable glaucoma (α) and sensitivity at 95% specificity (β). However, O2 do not contributed to the four final results as long as O2 is a boolean.

Second. When the image is indeed ungradable (the groundtruth is ungradable), my produced outputs for O1 and O2 are ignored, even if my O3 is False (Gradable). In other words, α and β are not influenced by my produced outputs O3 and O4. They are calculated on the partial image set in which images are indeed ungradable.

Third. γ and δ are calculated on the whole image set and are not influenced by my produced outputs O1 and O2,

Best Regards, He Wang

Hi wangh15,

That's all correct, except one thing. Assuming you refer to α and β with "They" in "They are calculated on the partial image set in which images are indeed ungradable.", it should instead be "They are calculated on the partial image set in which images are indeed gradable."

Best regards, Coen de Vente