Slide badly out of focus

Hi. The slide 61_HE_val is 100% out of focus. Was this done on purpose to see the registration algorithm is robust ? How are you able to annotate it without missing by thousands of microns ?

Hi,

thank you for your comment!

The annotators left a comment that the slide is out of focus and that landmark quality will be very poor. We know of further issues between image pairs, which are largely (but not exclusively) due to far apart sections. We decided to not exclude any WSIs for three reasons:

1. Annotator comments may not be comparable - we work with over 10 annotators, and it is unclear how well alligned the annotator perceptions for out of focus, far apart sections etc. are. It is therefore difficult to make objective decisions based on the annotator feedback.
2. Manual quality assessment would be possible for a data set of the size of the validation or test data set for this challenge. For larger data sets however, this will not be possible and we would like to emulate a realistic application of the algorithms as if they would be applied at scale in thousands of image pairs. (Although admittedly out-of-focus detection could be applied automatically.)
3. We would like to assess robustness of algorithms in the challenge-associated publication.

Since we chose the median of image pair scores as the summary score, image pairs with poor quality should not have a strong (or potentially any) influence on the challenge leaderboard.

Best wishes Philippe Weitz, Leslie Solorzano, Masi Valkonen

Hello, is the annotation of the 61st image pair in the validation set has an error? Because we noticed that even after we align the image pair well, the submitted results still show a large alignment error. Visually, the thumbnail image shows that the pair IHC is rotated 90 degrees clockwise with respect to HE, but we still have a large error after turning it around.

Hello, we seem to verify that the HE and KI67 of the 61st image pair in the set appear to be mirror images based on observations?

Hi,

thank you for your comment! We will look into this further. For now, we believe that is is not problematic to leave the image pair included since the median of the 90th percentile of distances (the chosen ranking metric) is unlikely to be impacted by a single poor image pair.

Best wishes Philippe Weitz, Masi Valkonen, Leslie Solorzano