Hello Luuk,
I have some questions regarding the final phase:
- What is the criteria to be qualified ? It is mentionned The best-performing teams will be invited to participate in the final round, what does it mean ?
- How many training runs are we allowed to perform during this phase ?
- It is mentionned that we will receive results of [our] algorithm on the validation fold of the public training data. Did you mean the 200 patients from the test set ?
- Is it going to be possible to access to the logs of the training runs ? Among the 200 patients of the public test set, only 118 are covid positive with 30 severe cases (see previous post here). The AUC on this small subset has a high variance* and might not be the best criteria to the select a model. Accessing the logs of the training runs would permit to access to cross-validation performances on 9,000 patients. In addition, training if much more difficult than inference and accessing the logs will certainly be helpful for debugging.
- How many solutions will be selected for the final rankings ? One per participant / team ? All the proposed solutions ?
Thanks,
Simon
*Following this paper, an upper bound of the variance is AUC * (1 - AUC) / min(n-positives, n-negatives). If we take AUC = 0.80 we get σ-max =7.3% for the 200 patients of the leaderboard test set, and σ-max = 1.1% for the 9,000 patients of the training set (supposing 15% are severe cases).
