Can you provide the test set of the qualification phase after the competition finishes?

We meet a problem. The train set was randomly splited into 3 parts, 1200 for train, 400 for validation and 400 for test. We trained our model under different hyper parameters. We saved the best model when it reached the best validation auc. Then we used the best model to predict the test set and got the test auc. The results are shown here:

trial 1: auc_val = 0.8178, auc_test = 0.8179 tiral 2: auc_val = 0.8026, auc_test = 0.8144 trial 3: auc_val = 0.8189, auc_test = 0.8105 trial 4: auc_val = 0.8084, auc_test = 0.8079 trial 5: auc_val = 0.8118, auc_test = 0.8193

After that, we did 5 folds cross validation. Each fold had 1600 images for train and 400 for validation. The validation auc of 5 folds were 0.7913, 0.7819, 0.8306, 0.8284, 0.8689, respectively. The average auc was 0.8202.

When we submited the algorithm, we just calculated the average value of the 5 predicitons . But the auc of the test set was only 0.75.

According to the previous 5 trials, the validation auc and the test auc are very close. And model ensemble usually gives a better result than one model. So we don't understand what happend. Does the test set have the same distribution as the train set? The age, gender and spacing are correct in test set? Can you provide the test set of the qualification phase after the competition finishes? So we can study what happend in the test set.

Hello,

The AUC displayed on the leaderboard is computed only among COVID positive patients (see discussion here). It may explain the gap.

Best, Simon

Hello Simon.

You are right. I calculated auc among COVID positive patients. The auc were decreased about 0.07. And I also checked the challenge overview: the focus of the challenge is the prediction of COVID19 severity at one month (AUC computed with COVID19 positive patients only, primary metric).

Thank you!

Hi,

The test set is a random subset of the full STOIC cohort (10,000+ patients). Any other data from the STOIC study than the public training data (2000 scans) cannot be publicly released. I hope this answers your questions,

Best, Luuk

Hi,

Just to check my understanding of the testing procedure.

" AUC computed with COVID19 positive patients only, primary metric" means that SEVERE/NOT-SEVERE AUC is estimated by applying first our algorithm for the COVID/NO-COVID classification, obtaining a COVID probability score, and considering the patient as "COVID" if outcome probability is >0.5. Just for patients classified as "COVID" with our algorithm the classification SEVERE/NOT-SEVERE will be performed.
All the cases classified as NO-COVID will be considered as NOT-SEVERE in the calculation of the final AUC.

Is this right?

Thank you for your timely support,

Marco

Hi, Marco,

I think "AUC computed with COVID19 positive patients only" uses the ground truth of the COVID19 positive patients, not the prediction of the algorithm.

Hi Marco,

The 'positive' in "AUC computed with COVID19 positive patients only" indeed refers to the PCR test outcome. I hope this clarifies things for you.

Best, Luuk

Thank you, So, I need further clarification.

• The "general test set" of the qualification round contains 1000 cases selected from the 2000 cases released for training (as described in a previous post of this discussion).
• The test set used for the estimate of SEVERE/NOT-SEVERE AUC includes just the COVID positive cases belonging to the "general test set" (therefore its size is lower than 1000)
• The test set used for the estimate of COVID/NOT-COVID AUC is the "general test set"

Are these points right?

Thank you again,

Marco

Hi Marco,

My previous post in this tread incorrectly stated "training set" where it should have stated "full STOIC cohort (10,000+ patients)". This has now been corrected. I hope this resolves the confusion.

• The "general test set" of the qualification round contains 1000 cases selected from the 2000 cases released for training (as described in a previous post of this discussion). The test set for the qualification round is a random subset of the full STOIC cohort (10,000+ patients).

• The test set used for the estimate of SEVERE/NOT-SEVERE AUC includes just the COVID positive cases belonging to the "general test set" (therefore its size is lower than 1000) The AUC for COVID19 severity is indeed only computed for COVID positive cases. This is the metric that is used to rank participants. The AUC for COVID19 presence is computed over all cases. This metric is only used as additional feedback.

• The test set used for the estimate of COVID/NOT-COVID AUC is the "general test set" The phrase "general test set" is not used in the challenge overview. Please refer to that page for more details.

I hope this clarifies things, Best, Luuk

Perfect!

Thank you Luuk