Create Your Own Challenge

This page contains the instructions for creating your own challenge on grand-challenge.org

Click here to host a challenge.

Most should be self-explanatory. You can also consult the docs.

Tools provided

We offer the following tools for challenge organizers:

  1. An easy way to create a site, add and edit pages like a wiki
  2. Registration mechanisms for participants
  3. Secure ways for organizers to provide challenge data to participants and for participants to upload results
  4. Automated evaluations of uploaded results
  5. Automated leaderboard management, including ways to tabulate, sort and visualize the results

Managing pages

You can add, edit, order, and delete pages for your challenge on the page navigation panel by selecting Admin -> Pages:

Automated evaluation

Every challenge has a unique way to objectively evaluate incoming submissions. More often than not, the evaluation scripts come with a set of dependencies and computational environments that are difficult to replicate on the host server. Therefore, we have decided that every challenge organizer has to provide a Docker that containerizes the evaluation scripts. This Docker will run on our servers to compute the evaluation scripts necessary for an incoming submission.

Building your evaluation container

To make the process easier, we created evalutils. It's a Python package that helps you create a project structure, load, and validate submissions and packages the evaluation scripts in a Docker container compatible with grand-challenge.org.

Requirements

  1. Docker
  2. evalutils

You can use your favorite Python environment to install evalutils.

pip install evalutils

Once you've installed the above requirements, you can follow the instructions for getting started and building your evaluation container here.

Uploading your evaluation container

Once you have created your evaluation container, you can upload it to your challenge by selecting Admin -> Methods

From this page, you can add and manage your evaluation methods.

NOTE: You can turn on automated evaluation by navigating to Admin -> Challenge Settings -> Automated Evaluation

Data storage

We use Zenodo to handle and store large datasets. The platform can be installed on your own server. We strongly recommend challenge organizers to use Zenodo. It has an upper limit of 50GB. We're currently working on other solutions for datasets larger than 50GB, feel free to write to us and we can help you.

Future plans

In 2020 we will start hosting challenges where participants can upload their algorithms in the form of Docker containers, which will be applied to the test data. This way of running challenges avoids that the test data is made available to challenge participants. We have also implemented the possibility to upload algorithms that users can try out with their own data, and web-based interactive browsers that can be used for reader studies. We plan a broader roll-out of this functionality in the near future.

Listing your challenge

If you just want to have your challenge listed on the overview on the Challenges page and you run your challenge on your own site, mail to support@grand-challenge.org with the details and we'll list your challenge.

Contribution

You are most welcome to help us further develop and extend the grand-challenge platform. You can use the bug/issue tracker and code repository on github, and create a new issue there.

About us

grand-challenges.org is currently maintained by the RSE Team of the Diagnostic Image Analysis Group in Radboudumc, Nijmegen, The Netherlands. More information about grand-challenges.org can be found here.

In 2012 a team with members from five groups in medical image analysis decided to build a platform to easily set up websites for challenges in biomedical image analysis. We named our group the Consortium for Open Medical Image Computing (COMIC). We decided the platform should be developed in Python and Django, and should be open source. grand-challenges.org runs on the hardware of the Diagnostic Image Analysis Group.