Tips for running a challenge
Test your submission process¶
Once you have everything set up, we advise you to test the submission process thoroughly before opening it up for participants. The Grand-Challenge support team is happy to help you debug any issues you face during this process.
Guide your participants to submission tips¶
Please guide your participants to the documentation presented here about submission tips.
Provide tutorials¶
It is also advisable to share an example algorithm script (including the Dockerfile and instructions on how to build the image) for your participants. You can use the challenge pack example that we share with you at the beginning of your set-up process as a baseline, but you will likely want to customize it a littler futher before sharing with your participants.
Walking participants through the submission process in a video tutorial can also help prevent questions. You can refer your participants to this section of the documentation or use the information there for your video tutorial. Examples from previous challenges can be found here and here.
Help your participants debug their submissions¶
We do not publish the submission logs openly on Grand-Challenge. Doing so could potentially leak information about the test data set. Debugging their submission is thus difficult for participants to do on their own. They will require your assistance. To inspect the logs, navigate to the Admin menu in your challenge and select 'Submission & Evaluations' from the phase specific tab menu. Click on the failed evaluation ID that you'd like to get more information about and scroll down to the 'Evaluation Admin' section. This will list the 'Prerequisite jobs' for this evaluation, and you can inspect them one by one by clicking on their IDs. You will then be able to see the logs that the algorithm produced. You can share the logs with the participants provided that they do not leak sensitive information about the test data. So please read the logs yourself before posting them.
Note that we do tell participants (as part of the submission tips that we advise you to publish in the forum) that they should thoroughly test their algorithm not only locally, but also on Grand Challenge. Once they have uploaded their container, they can try out their algorithm on test images of their choosing and even inspect the result in an online image viewer. Nonetheless, submission failures will happen so you need to know how to access the logs and help.
To aid in the initial submission development Phase you can make the phase have an open submission, providing increased access to the results by the challenge participant. This entails that the participants will automatically get access to their logs, the input data, and their predictions for that phase, saving you a potential lot of work.
⚠️ Warning: The consequence of this is that the data associated with that Phase then becomes accessible to the participants, so be careful when turning this setting on! We will email challenge admins and add a lot of warnings if this option is enabled.
If you would like to enable this setting please go to the Algorithm section of your phase settings and enable "Give Algorithm Editors Job View Permissions".
Optional: keep results hidden until a big reveal 🎊¶
By default, leaderboard rankings or submission details are updated every time someone submits to your challenge. An alternative is to keep them hidden until a specific day or event, making the publication a big reveal. Follow this workflow to enable such a big reveal:
1. Set Up the Phase:¶
Ensure Admin > Phase Settings > Scoring Tab > Auto publish new results is set to False.
2. Participant Submissions:¶
Allow participants to submit their algorithms. Note that participants will not receive their results immediately.
3. Close the Submission Phase:¶
Close the phase by either:
- Letting the submission date pass, or
- Setting the submission limit to zero.
4. Set Visibility to Private:¶
Change the phase visibility setting to hidden via Admin > Phase Settings > Public to False.
5. Publish Selected Evaluations:¶
Carefully publish the selected evaluations by manually pressing the 'Publish Result' button for evaluations listed in Admin > Phase Settings > Submissions & Evaluations.
6. Reveal Day:¶
On the designated reveal day, set Admin > Phase Settings > Public to True to make results visible to participants.