Prostate Validation Data - Did something change?

Hello,

I would like to ask about the validation sets (prostate but not only). Was the data or the ground-truth modified somehow in the last days?

I submitted exactly the same .csv file for the validation set as last time and got completely different results.

Edit: it seems that there is a randomization in the evaluation script - the same .csv file can produce different results after submission to the GC platform.

Bests,

Dear @IWM,

Can you tell us which submissions are supposed to be the same? In that way, we will be able to dive deeper into this. In principle, randomization is not applied during metrics calculation.

Thanks in advance. Best regards.

Hello,

The submission with ID: c16985c7-fec8-4dea-a10e-cc60ea95eeea and the submission with ID: e6e3794a-7df9-4af7-bb61-66aa0ff159c6

Bests,

Hi @IWM,

I have compared both submissions and the values provided for "10b0457ea00c792efd27ebd1504cd0422e8439f2955290ed0a83ed7fdf069580" subject are different.

Can this be the reason?

Best regards, Ana

Thanks, I checked that and it seems that a single case error (from random prediction into prediction with reasonable values) decreased the final score significantly. I also compared the best prediction (AUC=0.79) to the current prediction (AUC=0.63) and it seems that minor differences (at the level of 0.01 of the probability for majority of cases) result in significant changes in the final outcome. It is strange, however, possible.

Thanks for your time!