Phase 1 is extended until Thursday, April 04!

Hello everyone,

Thanks a lot to all participants that help us in highlighting issues.

We chose to extend the Phase 1 deadline to Thursday, April 04.
Do not hesitate to try submitting your docker containers (template available HERE)

By submitting to Phase 1 you will have access to your errors logs of your submissions (not allowed in Phase 2).
And the Phase 1 will not count towards the final assessment (only Phase 2).

Do not hesitate to contact us if needed,
Keep up the good work !

Hello,

The submissions appear to be closed for Phase 1. Also, is the code for evaluation metrics available yet?

It says: "Submissions are closed for this phase."

Also does "to April 04" mean the last possible submission is April 03 23:59 or April 04 23:59 since the last deadline was 30.03 and the submission is closed today

Hello everyone,

We're waiting to hear back from Grand Challenge as we seem to have run out of budget - which is not normal at all.
We can't do anything at the moment (for phase 2 too).
I'm really sorry, I'll let you know as soon as I have any news.

For the evaluation code, I've just put the link to the python file at the end of the Evaluation & Metrics section.
You can access directly it here https://seafile.lirmm.fr/f/8a17128c73e7493cba47/

Sincerely,
Dorian

Looking forward to the re-opening of the docker submission. I wonder if it would be possible for the teams to submit at least once to the leaderboard to make sure the docker submission is valid, in case the submission platform is closed till April 4th. Thank you.

Phase 1 submissions are now available again !¶

Sorry for the inconvenience !

Is deadline 4-th April AoE?

Hi,

I have quick question about the provided evaluation script. What is the point of normalizing both the predicted and ground-truth images? Shouldn't the normalization be applied only to the ground-truth images? With the current version of the evaluation script it assumes that the normalization is not performed during training and the training is performed using the original intensity ranges (otherwise the normalization would be applied twice). With such an assumption it makes the normalization unnecessary at all.

Bests,

Edit: Since I assumed (and maybe also others) that the normalization will be applied only to the ground-truth images - is there an option that we will declare whether our final submission should be normalized (the predictions) or not? Otherwise e.g. my submissions are biased by double normalization (running percentile normalization on already percentile normalized images will increase the MAE / Euclidean Distance). Moreover, it is impossible to easily reverse percentile normalization, like e.g. with range-norm or z-norm - so it would require to retrain the model completely from scratch.

Hi pyatkovsky15022001,

The dealine is CET.

Hi IWM,

You're right, if you normalize during the training, you'll be penalized.
Nevertheless, several types of normalization can be performed by the participants, as well as output image predictions that don't have the same range than GT or input images.
That's why we've chosen to normalize GTs and predictions for the evaluation.

Indeed, but performing the normalization just during the evaluation does not make much sense - it does not change anything except making the absolute numbers a bit smaller. The whole idea for performing the normalization is to apply it during training and the original question (in separate thread) about applying normalization was directly asking about ground-truth normalization.

Moreover, I agree that participants can introduce any normalization to the input images, however, I disagree that participants can apply different kind of normalization to the ground-truth. Any normalization different from the one applied during evaluation will introduce bias in MAE and Euclidean Distance - therefore the training should be performed directly against the version of ground-truth that will be then used for evaluation. Current version of the evaluation script makes any normalization undesired, thus, there is no point in applying the normalization during evaluation at all.

Edit: Since I assume that the evaluation script is final and will not change - can we submit the ISBI 4-page papers basically without the results? It seems that all the ablations have to be recomputed.

Hi IWM,
We've checked the "double" percentile normalization and the difference with the "simple" one on the metrics is tiny, so you won't be penalized too much.
Best,
Dorian

Edit: yes you can submit to ISBI 4-page papers basically without the results

Ok, thank you for the quick verification. Still, I will modify the scripts to take into account that both prediction and ground-truth are normalized during evaluation.