Hidden validation/tuning cohort

Wil this 100 cases be made available after the open development phase for the participants? If yes, will labels also be given?

Also, does the open development phase end on Nov 10 now? Is the deadline pushed?

Hi Sakina,

Imaging and annotations for the Hidden Validation and Tuning Cohort (100 cases), and the Hidden Testing Cohort (1000 cases), will not be made available (at least within the next 5 years, I presume). Our goal is to reserve them for benchmarking prostate-AI models through the future, as they allow us to compute an unbiased estimate of performance in a truly blinded testing setting. For more details, you can refer to Item 5 (Lifecycle Type) of our preregistered study protocol.

Deadline for submissions to the Open Development Phase has been extended to 20 November 2022. You can find an overview of our timeline here.

Hope this helps.

Thank you so much for your quick reply. Is there anyway your organization could be requested help in future even after the challenge has ended to validate/benchamark prostate AI models developed?

Hi Sakina,

Even after the 2022 iteration of the challenge has ended, we will continue to accept (and highly encourage) AI algorithm submissions from researchers and companies alike. We will accept submissions on a per-application basis. And for each approved application, we will benchmark the corresponding model performance on the Hidden Testing Cohort (1000 cases) and display its results on a common public leaderboard. More details will follow after the end of the challenge.

As for tuning your model hyperparameters or estimating performance using the substantially smaller + weaker Hidden Validation and Tuning Cohort (100 cases), people can continue to use the Open Development Phase - Validation and Tuning Leaderboard, which will remain fully open (as it is now) indefinitely.

Hope this helps.