Meaning of NaNs in data

Hello again,

I was inspecting the csv with the labels, and I noticed there are some NaNs in the "Label G1" (274) and "Label G2" (1377) columns, which I find a bit weird, since each image is supposed to have been graded by two graders, and only in case of disagreement G3 would say something. What would be the meaning of these missing values?

The additional labels also show NaNs for G1 and G2, exactly 274 and 1377 for all categories, so I guess there is some relationshio, right? Like someone lost those labels?

Ah, and while I have you here, what is the meaning of "U" that sometimes G1 and G2 use in their opinion? Is it unknown, unsure, or something like that?

Thanks!

Hi agaldran,

Thanks for the question, your observation is indeed correct! We evaluated the performance of the graders periodically and occasionally had to remove a grader. In such a case, all labels were removed because they could not be trusted.

In general, these eyes were re-evaluated by other graders - but not if they were already reviewed by the third grader. After all, the third grader overruled the other graders, so the labels of graders 1 and 2 were no longer relevant.

The 'U' means Ungradable, which could be selected by graders if they thought the image's quality was too poor to judge the eye. We excluded all image with a final Ungradable label in this challenge, but you can still come across them for the graders.

Does that sufficiently addresses your question?

Best, Koen