Evaluation of Justification Performance on non-referrable images

Hello again,

I was wondering how exactly does the evaluation work for referrable cases in the Justification performance part. The Evaluation & Metrics section says:

For the referable glaucoma cases, the 10 additional labels will be compared against the additional labels produced by the algorithm

Now, imagine my algorithm predicts on a non-referrable image that it is referrable, and then it gives some positive predictions on the additional categories. This would count as a mistake for the Referral performance category.

My question is, would this also count as a mistake for the Justification Performance category? Or since the image is actually non-referrable, those extra positive predictions are automatically ignored?

Thanks,

Adrian

Hi agaldran,

Only the images that are labeled (by us!) as referable glaucoma will be evaluated on the additional labels. We will not consider the no referable glaucoma images for this evaluation.

Best, Koen