Multiple phases, multiple leaderboards

A typical challenge has two phases: a preliminary and a final test phase, but your challenge can have as many phases (and hence leaderboards) as you want. As a rule of thumb, you should have separate phases for separate tasks (e.g., segmentation vs. classification) and use separate phases to split your submission period into, for example, a qualification and a final test phase. A typical challenge has a preliminary test phase in which participants get familiar with the algorithm submission procedure before submitting to the final test phase.

Each phase has its own submission procedure and its own leaderboard. Grand Challenge offers two types of submissions: prediction submission and algorithm submission.

The default submission type is algorithm submission: here participants package and upload their inference algorithms to the platform directly. They are run on a secret test set provided by the challenge organizers and the predictions they generate are subsequently compared against a secret ground truth by means of a custom evaluation method (again provided by the challenge organizers).

Alternatively, you can also set-up a phase for prediction submission, where participants upload their algorithm's predictions as a .csv or .zip file. This submission type can make sense as a preliminary qualification phase or when it is not possible to package the algorithm in a container, but is generally discouraged, because it provides a less fair, less objective and less reproducible comparison of submissions than the algorithm submission procedure.

To add a phase, navigate to Admin and click on Add a new Phase in the sidebar under Phases. Above that you see the links to the settings pages for each of the phases you have set up. You can customize the submission and scoring procedure for a given phase, set start and end dates for the phase, and decide what the leaderboard looks like and how the results are displayed.

⚠️ Note that if you set start and end dates, participants will not be able to make submissions to this phase before / after the respective start / end dates. Note also that you are free to set only a start (for an open-ended phase) or only an end date, or leave both dates empty. If there are no dates set, participants will be able to make submissions as long as the submission limit is set to a value above 0.

Examples of challenges that have made use of multiple phases are MIDOG2021 , PAIP2021 and PICAI.