Updated evaluation + challenge extension to 26. Jan 2025 (final test phase deadline)

Hi everybody

I've just re-opened the live leaderboard phase. I appreciate your patience.

The following updates/changes have been made:

• Updated all annotations of the validation set after another quality check review (remove double annotations, etc.)

• One additional case was added to the validation set (now there are 9)

• Changed the JSON files of the ground truth to be in mm instead of pixel (also available on AWS now) and updated the evaluation code accordingly (pushed to GitHub)

• Based on feedback from our pathologist, we changed the radius for a positive match to 5μm for monocytes and inflammatory cells. Even though the monocytes are larger, only the nucleus is clearly visible in the PAS staining, which is much smaller. This will most likely cause a slight drop in the performance of your existing algorithms.

• Added output for FROC curve plotting using the metrics.json file (code is provided on GitHub).

I requested a re-evaluation of all your existing submissions. I'll update this post when it's finished.

Additionally, I am happy to announce that we are extending the challenge phase into January 2025! The final test phase deadline will be the 26. January 2025 (the live leaderboard phase will remain open until then, too). You can make two different submissions to the final test phase, which will open on January 13, 2025.

Your latest submissions have been re-run. Please check if yours shows on the leaderboard (all old submissions are hidden). If there are any issues, please let me know!

Thanks for your contribution of this challenge. I re-downloaded the annotation folder but the ground-truth json files look unchanged. Where can I download the changed annotation files?

Hi! Yes, I just saw that the update was not rolled out to the AWS open data registry. I've sent a reminder to the relevant people. It should hopefully be out within the next few business days. There will be two subfolders in annotations: json_pixel and json_mm. Apologies for the delay!

The JSON files have been updated. If they are not up yet, they should be with the next AWS Registry rollout.

Can you please update xml folder as well.

The xml files are up to date, not sure what you mean? They have to be in pixels to be able to be read by ASAP.

Sorry for misunderstanding here but quoting your statement here "Changed the JSON files of the ground truth to be in mm instead of pixel (also available on AWS now)" So json files have just been converted from pixel to mm. Annotations haven't been changed u meant that? In that case xml files need not be touched. (I was bit confused as xml folder downloaded yesterday and previous downloaded version looked same)

Yes, I only added the JSON files converted to mm because the updated evaluation runs on these (it's also more consistent as the output from GC is also in mm). There are no changes to the training annotations, only the evaluation annotations. Hope that clears it up :)!