Zero DICE score in case

Hi, can you please check if DICE score for case 94 was calculated correctly in our submission ? Because in Slicer we see that tumor was found partially, but in the result JSON on leaderboard DICE for this class is 0.0 . Thanks

Additionally we also noticed different scores on other cases so you probably changed also calculation logic too (we noticed post about bug in averaging, but the calculation per case was ok before right ? )

Hi, after carefully checking, we confirm that our evaluation method is correct.

As for your second question, we did not change the calculation logic. The evaluation code has a technical vulnerability in our previous version, and in this version, we have corrected it.

Yes but how is possible that dice is 0.0 but other metrics have reasonable numbers. Also we compared submitted prediction with prediction from another model and dice was higher than 0. Can I email you prediction of another model just to be sure that calculation is correct, because we strongly believe that dice cannot be 0.0 in that case, also in Slicer we can see that tumor is marked on several slices. Thanks for answer.

Also regarding my second question:

You said that there was no change to calculation logic. But how is then possible that for the same submission zip file, after resubmission we got different scores on cases ? For me that does not make sense, something must have changed in your logic (besides averaging mechanism). Thanks for your patience.

Dear participant,

Thanks for your question which will encourage us to orangize our KiPA22 better.

For the first question, Please forgive us that we are unable to provide any additional information about the test results for another model. In order to be fair, the organizers are unable to do anything that will influence the judgment of the participants. But we have carefully checked this "zero DICE score" case visually, the predicted tumor is really not covering any real tumor regions, so its DICE is zero. The AVD and the HD are distance-based metrics, so if the result has no covered regions, these two metrics are still able to be calculated.

For the second question, the previous technical flaw which have been corrected is the mismatch between the structures and their evaluation score. You can find the zero dice score in this case is the "Vein" in your previous submittion. The calculation logic is still following the formulas in the evaluation page.

Yours, KiPA22 organizers.