Clarification on tumor volume annotation count in evaluation metrics

Dear organizers,

I had a question regarding the volume-distribution step in the evaluation metrics. In the description, you mention using three annotations to define the tumor-volume distribution, but our training data includes five expert segmentations per case.

Will the test set include only three annotations per case?

If so, how are those three chosen—are they a fixed subset of the five training raters (e.g. removing two), or a different selection entirely? Or should the evaluation actually use all five annotations for volume distribution?

Thank you for your help clarifying this detail!

Thanks for your message, and sorry for the confusion.

All metrics will be computed based on the five annotations provided (the original PANORAMA label plus four additional expert segmentations).

In the latest dataset release, you’ll also find a STAPLE-derived label. You're welcome to use it for training if you’d like, but it's mainly provided to help you test your metrics—since one of the two Dice scores in the evaluation will be calculated using this STAPLE annotation.

Best regards, The Organizing Team

Thank you for the clarification!