Create your own Challenge


If you are interested in hosting your own challenge on our platform, please carefully read this page and fill in our challenge request form. Since our resources are limited, we need to be selective about which challenges we host on Grand Challenge. Your answers to the request form help us make an informed decision about whether and how we can support your challenge.

When filling in the form, you will need to decide whether you want to host a Type 1 or a Type 2 challenge. You can read more about the differences between challenge types here, and about the respective set-up options here and here. In the spirit of fair and reproducible science, we highly recommend running a Type 2 challenge whenever possible.

You will also be asked to provide an acronym for your challenge in the request form. We will use this for the url of your challenge (e.g., https://{acronym}.grand-challenge.org/) if your challenge gets accepted. It will also be used for specific css and files. For this short name, no special characters or spaces are allowed.

If you pre-registered your challenge on the BIAS website you have the option to upload your submission PDF and fill in the text fields in our form with "See PDF".

From 📃 challenge request to 🚀 challenge launch

After submitting the form, you will receive a confirmation email. If you requested a type 2 challenge, this email will also contain the budget estimate for your challenge based on the specs you entered.

Our team of reviewers will then evaluate your submission and inform you of the decision within maximally 4 weeks (we strive to inform everyone within 2 weeks, but there are times when that is not possible). If your challenge is accepted, we will create the challenge page for you and share the link to it with you in the acceptance email. The challenge will initially be hidden, meaning that it will not yet be displayed on our challenge overview page.

You can then proceed to:

Once your challenge is ready for the public, you can change its status from hidden to public.

If your challenge gets rejected, you will also be notified by email.

The general workflow of requesting a challenge is summarized below. Visit the type 1 and type 2 challenge setup pages for a more detailed explanation on how to proceed after your challenge has been accepted. If you have any questions about the request procedure, or experience difficulties in setting up your challenge afterwards, do not hesitate to contact our support team at support@grand-challenge.org


Budget planning for Type 2 challenges

For Type 2 challenges, the request form contains questions about the size of the test data set, the number of submissions you plan to accept from participants and the average time you expect an algorithm run (including model loading, i/o, preprocessing and inference) to take per image. This information will be used to calculate a rough budget estimate, which in turn will help us decide whether we can finance your challenge, or whether we need to make other arrangements. You also get the chance to indicate whether or not you have funding avaiable to run your challenge on our platform.

To make sure you fill in the budget fields as accurately as possible, we have collected a few example budgets here.

Example of a type 2 challenge with 1 task

In the simplest case, a type 2 challenge has 1 task and is carried out in 2 phases. The first phase is usually a preliminary phase where participants familiarize themselves with the submission system and test their algorithms on a small subset of images. The second phase is the final test phase, often with a single submission policy, which evaluates the submitted algorithms on a larger test set. You could also think of the two phases as a qualification and a final phase, where you use the qualification phase to select participants for the second, final test phase. The definition is up to you. For the budget estimate, we simply assume that there will be two phases for a type 2 challenge and hence ask for estimates for two phases separately.

The MIDOG 2021 challenge is an example of such a challenge. For the MIDOG challenge, participants (~50 teams) had to develop algorithms to detect mitotic figures in histological tumor images (average size of image ~ 150 MB). The challenge consisted of two phases: a preliminary phase for participants to test their algorithms ( ~15 submissions per team) on a small subset of images (N=20), and a final test phase (1 submission per team) which evaluated the algorithms on a larger test set (N=80). The submitted algorithms took an average of 5 minutes for inference per test image.

The budget for the MIDOG challenge amounted roughly to the following:

Cost Amount
Compute costs for phase 1 (preliminary phase): 1250,00 $
Storage costs for phase 1 (preliminary phase): 0,12 $
Compute costs for phase 2 (final test): 333,33 $
Storage costs for phase 2 (final test): 0,48 $
Docker storage costs: 200,00 $
Total: 1783,93 $


Example of a challenge that uses batched images as input

Grand Challenge runs the submitted algorithms on each image of the test set you provide separately. Each algorithm job thus requires loading of the model from scratch before inference can be run. This means that if you have a lot of test images, running the submitted algorithms is going to be very costly. What you can do to circumvent this and to reduce the overhead is divide your test set into batches of images and stack them into .tiff or .mha images. This drastically reduces the number of "images" in your archive and hence the number of algorithm jobs that will be initiated by our platform for each submission. You could even stack all test images into one file, bear in mind though that a single algorithm job cannot take longer than 2 hours, thus putting a limit to how many images you can realistically stack into one file for inference in one job.

The organizers of the Airogs challenge went for this solution: instead of uploading 11.400 single test images, they created batches of 300 images, stacked into tiff files, resulting in 38 test files. For their budget estimate, they then simply entered the number of batches (not the number of single images) and provided the size of the batch (rather than the size of a single image):

  • Size of test image: 300 MB (size of the batch file, i.e., 1 test image here contains 300 images stacked)
  • Number of test images for final phase: 38 (i.e., 38 batches)
  • Number of test images for preliminary phase: 4 (i.e., 4 batches)

They expected roughly 50 teams and an inference time of about 55 minutes per batch (! not per image this time) and they allowed 3 submissions per team to the preliminary phase and 1 submission per team to the final test phase. Their cost estimate was as follows:

Cost Amount
Compute costs for phase 1 (preliminary phase): 183,33 $
Storage costs for phase 1 (preliminary phase): 0,05 $
Compute costs for phase 2 (final test): 5225,00 $
Storage costs for phase 2 (final test): 0,45 $
Docker storage costs: 200,00 $
Total: 5608,84 $


Example of a challenge with 2 tasks:

If your challenge deviates from the standard format of 1 task, 2 phases, please indicate reasonable average numbers across all tasks/phases of your challenge. The Conic challenge, for example, had two tasks: participants had to develop algorithms to (1) segment and classify nuclei within tissue samples and (2) predict how many nuclei of each class are present in a given input image. For each task, there were 2 phases (a preliminary phase and a test phase).

For a budget estimate for such a challenge, provide the averages across the two tasks for all the fields respectively and indicate that your challenge has 2 rather than just 1 task:

  • Number of tasks: 2
  • Size of test image: (size of test image for task 1 + size of test images for task 2) / 2
  • Inference time: (inference time task 1 + inference time task 2) / 2
  • Number of submissions for phase 1: (N submissions phase 1 task 1 + N submissions phase 1 task 2) / 2
  • Number of submissions for phase 2: (N submissions phase 2 task 1 + N submissions phase 2 task 2) / 2
  • Number of test images phase 1: (N images phase 1 task 1 + N images phase 1 task 2) / 2
  • Number of test images phase 2: (N images phase 2 task 1 + N images phase 2 task 2) / 2


Example of a challenge with more than 2 phases:

If your challenge has 1 task but more than 2 phases, please provide the average number of submissions across all phases and the sum (!) of test images across all phases in the two fields for phase 1 and enter 0s for the phase 2 fields:

  • Number of tasks: 1
  • Size of test image: average across all phases
  • inference time: average across all phases
  • Number of submissions for phase 1: average across all phases
  • Number of submissions for phase 2: 0
  • Number of test images phase 1: sum of all test images across phases
  • Number of test images phase 2: 0