Make a Challenge Request¶

If you are interested in hosting your own challenge on our platform, please carefully read the challenge policy and pricing and running a challenge on grand challenge pages first, to ensure we can offer you the services you require to run your challenge. If you have any questions or wish to deviate from the standard policies, don’t hesitate to contact us via support@grand-challenge.org.

📃 Request your challenge¶

To start the process of requesting a challenge, please fill in the challenge request form.

You will be asked to provide an acronym for your challenge. This acronym will:

• form the basis of your challenge URL (e.g., https://{acronym}.grand-challenge.org/)
• be used in file naming and CSS configuration.

Note: The acronym must contain no spaces or special characters.

If you pre-registered your challenge on the BIAS website you have the option to upload your submission PDF and fill in the text fields in our form with "See structured submission form".

Compute and storage cost estimate¶

The form will ask for technical details to help us estimate the cost of running your challenge. These include:

• Size of the test dataset
• Expected number of participant submissions
• Average inference time per image (including model loading, I/O, preprocessing, and inference)
• GPU type and memory requirements for each submission's runtime environment

Our team will review these details and work with you during onboarding to finalize a cost-effective plan.

For examples of how to fill in these fields, see our example setups.

From challenge request 📃 to challenge launch 🚀¶

After submitting the challenge request form, you will receive a confirmation email. Our review team will then evaluate your request and notify you of the outcome within 4 weeks.

The evaluation is based on the following criteria:

1. Scientific Soundness and Relevance
   We assess the scientific quality, novelty, and impact of your challenge.

2. Technical Feasibility and Platform Compatibility
   We verify whether your technical requirements — such as data formats and container setup — are supported by the Grand Challenge platform.

3. Cost Estimation and Feasibility
   Based on the numbers you provide in your form, we will generate a budget estimate. If needed, we will work with you to find ways to reduce costs and arrive at an agreeable budget.

If your challenge is rejected, you will be notified via email with a brief explanation.

If your challenge is accepted:

• We will create the challenge page for you
• You will receive a link to the page in your acceptance email
• You can then continue to set up your challenge with support from our team

If you have any questions about the request procedure, or experience difficulties in setting up your challenge afterwards, do not hesitate to contact our support team at support@grand-challenge.org

Examples of phase and tasks setups¶

📌 Standard Challenge with 1 Task, 2 Phases¶

In the simplest case, a challenge has 1 task and is carried out in 2 phases. The first phase should be a debugging phase where participants familiarize themselves with the submission system and test their algorithms on a small subset of images. The data and the inference results for this phase should be open to allow the participants to debug their own containers on the platform. The ranking of the submissions to this phase should be meaningless. It is advisable to add the cases that would require the most memory and/or compute time in this phase to ensure the inference is correctly implemented for the final, hidden phase.

The second phase is the final test phase, often with a single submission policy, which evaluates the submitted algorithms on the hidden test set.

You could also think of the two phases as a qualification and a final phase, where you use the qualification phase to select participants for the second, final test phase. The definition is up to you. For the cost estimate, we simply assume that there will be two algorithm submission phases and hence ask for estimates for two phases separately.

📌 Challenge using Batched Image Inputs¶

Grand Challenge runs the submitted algorithms on each image of the test set you provide separately. Each algorithm job requires loading of the model from scratch before inference can be run. This means that if you have a lot of test images, running the submitted algorithms is going to be very costly. If you have a large number of images (>1000) that are each small in size, what you can do to circumvent this and to reduce the overhead is divide your test set into batches of images and stack them into .tiff or .mha images. This drastically reduces the number of "images" in your archive and hence the number of algorithm jobs that will be initiated by our platform for each submission. Note, however, that stacking images comes with the downside of not being able to visualize the algorithm results in our online viewer. The viewer is not designed to handle stacked images.

📌 Challenge with 2 Tasks, 2 Phases¶

If your challenge deviates from the standard format of 1 task, 2 phases, please indicate reasonable averages across all tasks/phases of your challenge. The Conic challenge, for example, had two tasks: participants had to develop algorithms to (1) segment and classify nuclei within tissue samples and (2) predict how many nuclei of each class are present in a given input image. For each task, there were 2 phases (a preliminary phase and a test phase).

For a budget estimate for such a challenge, provide the averages across the two tasks for all the fields respectively and indicate that your challenge has 2 rather than just 1 task:

• Number of tasks: 2
• Size of test image: (size of test image for task 1 + size of test image for task 2) / 2
• Inference time: (inference time task 1 + inference time task 2) / 2
• Number of submissions for phase 1: (N submissions phase 1 task 1 + N submissions phase 1 task 2) / 2
• Number of submissions for phase 2: (N submissions phase 2 task 1 + N submissions phase 2 task 2) / 2
• Number of test images phase 1: (N images phase 1 task 1 + N images phase 1 task 2) / 2
• Number of test images phase 2: (N images phase 2 task 1 + N images phase 2 task 2) / 2

📌 Challenge with more than 2 Phases¶

If your challenge has 1 task but more than 2 phases, please provide the average number of submissions across all phases and the sum (!) of test images across all phases in the two fields for phase 1 and enter 0s for the phase 2 fields:

• Number of tasks: 1
• Size of test image: average across all phases
• Inference time: average across all phases
• Number of submissions for phase 1: average across all phases
• Number of submissions for phase 2: 0
• Number of test images phase 1: sum of all test images across phases
• Number of test images phase 2: 0