Challenge Proposals MICCAI
For reviewing Challenge proposals for MICCAI 2015, we will pay attention to the following
- Relevance: Does the challenge address an appropriate and relevant task? A problem that only very few groups are addressing may not (yet) be a good idea for a competition, as there are simply not enough potential participants (yet). A problem that is more or less solved is also not a good idea. Challenges ideally address an important open problem for which some solutions are available, so that the time is right for a fair and direct comparison of different approaches.
- Data Quality: It is important to provide enough data, and this data should contain enough variability to be representative for the problem. In general, it is preferred to have data from different scanners or devices, obtained with different protocols or workflows as used in clinical practice worldwide, and from different institutions or populations. A typical limitation of published papers is that a proposed algorithm is evaluated on data from only a single site. A good challenge does not have this limitation.
- Training Data: Prospective participants to challenges often appreciate the availability of ample training data. Provide this data if possible. Of course, training data should be representative of testing data. Please indicate in the rules of your challenge how participants should use this training data, i.e. are they allowed their own training data as well or not, or are there different tracks depending on what training data was used.
- Test data: Make sure you properly separate training and test data. It is preferred to include some test data from protocols/scanners/institutions that are not represented in the training set.
- Reference Standard: The methods of defining the reference standard and of evaluating algorithm results must be clearly defined and generally agreeable to the academic community. The challenge is unlikely to attract interest from serious contenders if these methods are poorly considered or open to question. For a paper describing procedures to define reference standards see http://www.hal.inserm.fr/file/index/docid/185431/filename/Jannin_Manuscript-revised3IJCARS2007.pdf
- Participants: The details of the challenge should be well publicized and advertized in the relevant circles in order to attract a reasonable number of participants, without which the final results will be of less interest. Collecting all prior work relevant to the task at hand and personally inviting the authors of this work has been shown to be a good procedure. In your challenge proposal, please include an overview of key papers relevant to your challenge. A list of prospective participants is also appreciated, or otherwise an estimate on the number of expected participants. Include your plans for how to attract participants in your proposal.
- Organizing Team: Diversity in the team of challenge organizers is generally recommended. It is good to include a number of people from different backgrounds with experience in the field. These might include researchers from a number of different academic institutes, as well as from industry, who have worked on a variety of projects related to the topic of interest. This will ensure access to a larger pool of data and contacts, as well as a balanced set of opinions on how to define the reference standards and evaluate algorithm performance.
- Website: A good challenge should not end with a workshop at a conference. Make sure you include a high quality website with your challenge, set up in such a way that for many years to come, new submissions can be processed quickly and efficiently. Using a framework for hosting your challenge and/or your data is recommended. Suggestions are provided here.
- Open access: A challenge is more attractive to the field if the data and the algorithms are open, or will be open, or publicly available, also after the challenge event. Please include a paragraph on how you deal with availability of data and algorithms in your proposal.
- Publications: A high profile overview paper on the results of your challenge is a good way to inform the community about the results of your comparative study. Include your plans on this topic.
Bram van Ginneken
Valeria De Luca