Submissions to the Open Development Phase - Testing Phase

Hi everyone.

Once you have gone through your whole AI model development + tuning cycle, and are now certain that you have your final algorithm ready and trained for independent testing, please complete the following steps:

• Make one final submission to the Open Development Phase - Validation and Tuning Leaderboard following all the steps stated here (only if you haven't already). This is to ensure that your final model is bug-free and performs just as expected.

• Optionally, you can prepare a document (in PDF format; minimum 1 page; no restrictions on template or maximum length), briefly covering the model design and training strategy of your AI algorithm. Please include the names, designations and contact details of your team members in this document. You can upload this document using the "Supplementary File" field in the submission form, or using the "Publication" field to share a URL to the same. If you're submitting an AI algorithm that has already been presented in one of your team's prior publications, then you can directly share a URL to that publication. Please note, this step is mandatory if you want to qualify as one of the top 5 teams of the challenge and move on to the Closed Testing Phase. If you have been selected as one of the top 5 algorithms and have not provided this document in your submission, we will contact you for the same later down the line.

• Navigate to the "Open Development Phase - Testing Submission" page, select your final AI algorithm, optionally upload your supplementary file and/or enter your publication URL, and then click "Save" to submit. Please note, each team (not each participant per team) can only submit a single AI algorithm to the Open Development Phase - Testing Leaderboard. If multiple submissions are made, only the last submission from that team at the end of the submission deadline will be used to infer performance on the Hidden Testing Cohort.

• Please send us an e-mail at anindya.shaha@radboudumc.nl and joeran.bosma@radboudumc.nl, with your team name, list of team members, and a URL to your submitted algorithm (here is an example). We will use this e-mail thread to share updates regarding your submission and address issues (if any). Additionally, if you make it to the top 5, the designated team will receive the prizes listed here.

Results of all submissions (that are error-free and compliant with all participation rules) will be pre-computed in a hidden setting, and presented during our oral presentation at RSNA 2022. After this presentation, the results will also be made publicly available on the leaderboard.

We will begin accepting submissions to the Open Development Phase - Testing Leaderboard on 15 November 2022, and close the submissions page on 20 November 2022.

[UPDATE AS OF 21 NOVEMBER] We have extended the deadline for submissions to the Open Development Phase - Testing Leaderboard by 48 hours (2 days) to 22 November 2022. This extension is only meant to support failed/incomplete submissions, that couldn't be adequately resolved through the weekend.

Hope this helps.

All the best!

Regarding your first point, "Make one final submission to the Open Development Phase - Validation and Tuning Leaderboard following all the steps stated here (only if you haven't already). This is to ensure that your final model is bug-free and performs just as expected."

Is is okay to assume it meets the requirements if it has passed submission for the validation/tuning leaderboard?

Thank you!

Hi Sakina,

If that exact same algorithm (same Docker image) had passed submission for the Open Development Phase - Validation and Tuning Leaderboard once before, then it's okay to assume that it meets the requirements.

Hope this helps.

Perfect, that's all I wanted to clarify. Thank you!

Hi,

I submitted one of my algorithm that had succeeded for the validation phase for the testing phase. But now it is showing "Failed - The algorithm failed on one or more cases." for the open testing phase. Could you please help me with this? Thank you!

Best regards

Hi Sakina,

I've requested access to your algorithm. Once you've accepted, I can have a look and revert back to you via e-mail.

Hope this helps.

We have extended the deadline for submissions to the Open Development Phase - Testing Leaderboard by 48 hours (2 days) to 22 November 2022. This extension is only meant to support failed/incomplete submissions, that couldn't be adequately resolved through the weekend.

Hope this helps.