Create Your Own Challenge¶

Running a Challenge on grand-challenge.org¶

Challenges on grand-challenge.org are designed to objectively evaluate submitted algorithms in a secure and reproducible environment. A core principle of the platform is that both the test data and the test labels remain hidden from participants at all times.

Participants submit an algorithm to solve the task defined in your challenge. These algorithms are automatically executed on the hidden test dataset, which must be uploaded to an archive by the challenge administrators. After the algorithm has run on the test data, the output it produces is evaluated using a custom evaluation method that you provide.

The evaluation generates performance metrics, which are:

• Automatically calculated after each submission
• Displayed on the leaderboard
• Used to rank participants based on defined criteria

All submitted algorithm and evaluation containers are executed on our scalable AWS infrastructure, which adjusts compute and storage capacity automatically to meet demand.

For each phase of your challenge, submitted algorithms are run on every item in the linked archive. A runtime environment is selected based on:

• The resources you allow for that phase
• The requirements declared by the participant in their algorithm submission

🔐 Submitted algorithms do not have access to the internet, and participants (by default) do not see logs or intermediate results. This protects the integrity of your test data and prevents data leakage.

As a challenge admin:

• You have access to logs and results from each submission
• You do not have access to the algorithm code itself, unless the participant explicitly shares it

Challenge Structure¶

A typical challenge on the platform consists of two phases:

1. Preliminary Phase
   This is a warm-up or validation phase where participants can:

   • Learn how to submit algorithms
   • Test their submissions on a small dataset
   • Ensure compatibility with the platform and your evaluation setup

   This phase is highly recommended, as most participants require several attempts to create a valid submission.

2. Final Test Phase
   In this phase:

   • Algorithms are evaluated on the full hidden test dataset
   • Submission limits are often stricter (e.g., one final submission)
   • Rankings are used for final evaluation or awards

You can also structure your challenge with additional phases, such as a qualification phase to filter participants for a final round, or support multiple tasks or modalities using interfaces.