Update on Final Test Phase Info ❗️❗️❗️

Dear participants,

We would like to give you 2 updates:

1) We have found for task 1, the PQ per class in the leaderboard and the output logs do not correspond to the correct nuclei type. This does not affect the final mPQ+ result, but we should inform you in case you were optimising for a particular class. The correct class mapping is as follows:

(type on leaderboard: actual type) Epithelial: Neutrophil Neutrophil: Plasma Lymphocyte: Epithelial Plasma: Lymphocyte Eosinophil: Eosinophil Connective: Connective

Note, for task 2, the leaderboard stats per class is correct!

2) The maximum run time allowed for the final test phase is 60 minutes. Note, this is for a single input array of size 1010x256x256x3 (1010 images). If your submission does not pass (but it did pass in the preliminary phase), then this will be because it exceeded the max time limit. You should ensure that you test how long your docker container takes to run before submitting. If you can't get your solution to run under 60 minutes - an alternative is to send across your Docker container and we can run your container locally - but first please try to see whether your submission passes under the time limit. However, if doing this, your algorithm MUST have passed during the preliminary test phase. We will provide more information on this very soon, but please note that this should only be done as a last resort!

Due to the above, we are going to reopen the preliminary test phase for an extra 2 days (closing on the 2nd March at 23:59 GMT). We will still keep the final test phase open and so you are still welcome to submit your algorithm when you are ready.

Good luck!

Team CoNIC 🚀

2022-03-01 Edited by vqdang: In case that you want to know your docker runtime for the preliminary test set, please fill your team name here , we will periodically check this form every 2h to add more information.

Hey Simon,

do the GPUs that process our submissions run in default compute mode oder exclusive compute mode?

Best Lorenz

"The maximum run time allowed for the final test phase is 60 minutes. Note, this is for a single input array of size 1010x256x256x3 (1010 images)."

This is a bit hard to test locally, since we don't have the same CPU/GPU as the compute environment that the Docker container runs in. It would help if we could know how long our preliminary test submissions took and how many images they processed.

Hi Lorenz,

We run in default mode and each inference container gets exclusive access to 1 GPU.

@uschmidt83,

We are going to collate some information for all teams based on your request. We are looking into it right away.

Best,

Simon

@uschmidt83, @lorenz.rumberger

You can put your team name here and we will check and provide your runtime for preliminary test set.

For ease of communication, we will provide the runtime of your most recent submission.

Best Regards,

Hi,

When we saw this announcement it is already March 1st, so we missed the chance for submission on Feb 28th. We can only submit once for today, but it is not so fair because other people would have one more extra chance to submit because of the time zone. Can we have one extra submission chances for Mar 1st?

Best, CY

We will increase the number of submissions per day to 3 per task until the end of prelim test - this should help you optimise your models for submission to the final test! You should only need to use this for optimising the run time.