Submitted algorithm evaluation time

Hello team,

Although we successfully submitted our docker more than 35 hours ago (status is still active), we dont see our results on leaderbord. Is that normal? How would we know if there is a problem or not?

Thanks Mert

Hi Mert,

I'm afraid that I don't see any submissions from you or your team to the PI-CAI challenge. Did you follow all the steps listed on https://pi-cai.grand-challenge.org/ai-algorithm-submissions/? Perhaps you've uploaded your algorithm container to Grand Challenge, but forgot to submit that algorithm to the Open Development Phase via https://pi-cai.grand-challenge.org/evaluation/open-development-phase/submissions/create/? Once this submission is made, your results are published to the leaderboard typically within 24 hours (often within 1-3 hours).

Hope this helps.

Hi,

Thank you, you are right. That was my faulth. Helps a lot.

Thanks

Hi Mert,

No worries. It seems that your algorithm submission failed. We highly recommend "trying out" your algorithm with a sample case first (you should see this option on the left pane of your algorithm page on Grand Challenge), before making a submission to the leaderboard. It should help you confirm that it's working, and if not, then what's the error message such that you can debug it. If it works with a sample case, then feel free to go ahead and make a submission. Do let us know, if you have any further queries/concerns.

Hope this helps.

Hi Anindo,

Yes, it failed. Exactly, this is the best way to find the error. Trying to fix it.

Thanks a lot Mert

Hi Anindo,

There is also one more thing to be clarified. What should "/output/cspca-case-level-likelihood.json" file include? Is that a single file contains case level probabilities of all cases? If yes, can you show an example for multiple case predictions?

Thanks Mert

Hi Mert,

Each instance of an algorithm container only processes a single case (imaging + clinical information) to produce a single set of outputs (csPCa detection map + case-level csPCa likelihood score) for that given case. During validation, e.g. on the Open Development Phase - Validation and Tuning Leaderboard, 100 instances of your algorithm container are executed for the 100 hidden validation cases independently, on a one-to-one basis. In other words, your algorithm container isn't meant to output the case-level probabilities of all cases, but rather the case-level probability (+ detection map) for a single given case. We highly recommend that you refer to the baseline algorithm templates provided by us and directly adopt its I/O operations for this. You can also pass a sample training case through one of our baseline algorithms on grand-challenge.org, e.g. the baseline nnU-Net (semi-supervised), to get an example set of expected outputs.

Hope this helps.

Hi Anindo,

Perfect, everything is clear now. Thank you so much.

Mert

Hi Mert,

Your team's recent submissions to the leaderboard failed, because your algorithm is generating csPCa detection maps that don't have the exact same shape as their input T2W images (which we mandate, as stated here). Please refer to Lines 174-196 of process.py in this repo (trained baseline nnU-Net GC algorithm), or Lines 238-270 of process.py in this repo (trained baseline U-Net GC algorithm), for examples on how this can be handled. If your output csPCa detection maps don't have the same shape as their input T2W images, then they won't have the same shape as their ground-truth annotations. In turn, evaluation on the leaderboard will fail from this shape mismatch.

Hope this helps.

Hi Anindo,

Our algorithm works properly on pre-submision step but explode in evaluation. We understand the problem which sometimes occur & fixed it. Thank you so much.

Mert

Hi Anindo,

Thank you so much for your assistance, you helped a lot. Normally we have 1 submission right per verified user per day but we could not do successfull submission. In that case could we do one more submission trial in the same day after fixing our algorithm?

Thanks Ahmet

Hi Mert and Ahmet,

Happy to help. No worries, you get one successful submission per verified user per day. Erroneous submissions do not count towards this limit, so you can keep trying.

All the best.

Hi @Anindo,

We have a similar problem with our recent docker submission. Our submision still showing as "Excuting Algorithm" for almost 12 hours. This never happend before and normally it takes 2-3 hours for algorithm evaluation.

Could you please check our recent submission?

Best, Sulaiman

Hi Sulaiman,

Noticed the same and agreed. It's pretty strange that it's just stuck like that. Looking into it with the support team @ grand-challenge.org. Might take some time. Meanwhile, if you want quick results, you can reupload that same Docker image to your algorithm once more and ask one of your teammates to make another submission (since your one submission per 24 hours is being used for the current submission).

Hope this helps.

Hi Anindo,

Sorry to bother you again. However, my submisisons failed two times. I tested locally and it works, also the code is still the same as our previous submission.

Could you please check what is the issue with our two recent submissions?

Best, Sulaiman