ℹ️ The Evaluation Phase is open! ✅

ℹ️ The Evaluation Phase is open! ✅
¶

After your mock tests in the Test Phase, you are can make a single submission in the Evaluation Phase, which will count as the final result to determine the final ranking. 🏁🔬🌐

🚀 The Evaluation Phase is open until March 17.⌛📅

Information about the Evaluation Database¶

Total : 32 images¶

• similar & new images distribution:

  • 27 images from the same studies as the training database (~85%)
  • 5 images from new (5) studies in similar acquisition condition (~15%)

• organelle distribution:

  • 12 membranes (with 10 segmentations)
  • 20 nucleus (with 3 segmentations)

Is it possible to have a test phase submission counter reset ? I would like to test the final method before sending it to evaluation.

"You are can make a single submission in the Evaluation Phase, which will count as the final result to determine the final ranking."

Checking the submission form, the following statement is written :

You can create 1 more submission in the next 5 days to this phase. Algorithm jobs created for this phase will have a time limit of an hour per case.

Is it possible to update our submission after some days, or is the evaluation submission single and definitive ?

Thanks!

Hello @cyril.meyer.68,

You should be able to submit again for the Test Phase (max twice). For the Evaluation Phase it is only one unique submission with final result indeed.

I will see on Monday to make a poll about the relevance of giving one more week for the evaluation phase.

Best,

During the evaluation phase, is submission result public ?

Currently, for the evaluation phase, the full content of the metrics file is available on the results detail page and via the API.

As the logs will only be seen by the organizers, I allow this to give you a better understanding of your results in the final submission.

If you don't think this is relevant, I can launch a poll asking all participants to display only the final scores publicly, or the entire content.

Hi everyone,

unfortunately, my image doesn't seem to run through on the evaluation data set (although it was working before on the test data). Would it be possible for you to share the log files to be able to debug it (or at least the relevant portion thereof :-) )? I couldn't find any information useful for debugging in the submitted results tables due to the failure.

Thanks in advance and best regards,

Johannes

Hello,

It seems that the images doesn't take into account the new studies (start from 6):

FileNotFoundError: [Errno 2] No such file or directory: 'resources/weights/Study9_Nucleus.ckpt'
FileNotFoundError: [Errno 2] No such file or directory: 'resources/weights/Study6_Membrane.ckpt'
FileNotFoundError: [Errno 2] No such file or directory: 'resources/weights/Study7_Nucleus.ckpt'
FileNotFoundError: [Errno 2] No such file or directory: 'resources/weights/Study10_Nucleus.ckpt'
FileNotFoundError: [Errno 2] No such file or directory: 'resources/weights/Study8_Nucleus.ckpt'

2025-03-10T18:05:27.868000+00:00 Traceback (most recent call last):
2025-03-10T18:05:27.868000+00:00   File "/opt/app/inference.py", line 226, in <module>
2025-03-10T18:05:27.868000+00:00     raise SystemExit(run())
2025-03-10T18:05:27.868000+00:00                      ^^^^^
2025-03-10T18:05:27.868000+00:00   File "/opt/app/inference.py", line 107, in run
2025-03-10T18:05:27.868000+00:00     image_predict = fmc_entry_point(input_name_csv, ckpt_path, in_channels, image_groups)
2025-03-10T18:05:27.868000+00:00                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-10T18:05:27.868000+00:00   File "/opt/app/resources/apply_script_docker.py", line 298, in fmc_entry_point
2025-03-10T18:05:27.868000+00:00     return fmc_process_image(hyperparams, network)
2025-03-10T18:05:27.868000+00:00            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-10T18:05:27.868000+00:00   File "/opt/app/resources/apply_script_docker.py", line 71, in fmc_process_image
2025-03-10T18:05:27.868000+00:00     model.load_pretrained(pretrained_file=hparams.ckpt_path)
2025-03-10T18:05:27.868000+00:00   File "/opt/app/resources/models/ImageFusionUNet3D.py", line 60, in load_pretrained
2025-03-10T18:05:27.868000+00:00     state_dict = torch.load(pretrained_file)['state_dict']
2025-03-10T18:05:27.868000+00:00                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-10T18:05:27.868000+00:00   File "/opt/conda/lib/python3.11/site-packages/torch/serialization.py", line 1425, in load
2025-03-10T18:05:27.868000+00:00     with _open_file_like(f, "rb") as opened_file:
2025-03-10T18:05:27.868000+00:00          ^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-10T18:05:27.868000+00:00   File "/opt/conda/lib/python3.11/site-packages/torch/serialization.py", line 751, in _open_file_like
2025-03-10T18:05:27.868000+00:00     return _open_file(name_or_buffer, mode)
2025-03-10T18:05:27.868000+00:00            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-10T18:05:27.868000+00:00   File "/opt/conda/lib/python3.11/site-packages/torch/serialization.py", line 732, in __init__
2025-03-10T18:05:27.868000+00:00     super().__init__(open(name, mode))
2025-03-10T18:05:27.868000+00:00                      ^^^^^^^^^^^^^^^^
2025-03-10T18:05:27.868000+00:00 FileNotFoundError: [Errno 2] No such file or directory: 'resources/weights/Study9_Nucleus.ckpt'

Don't hesitate if you need more information,

Best,

Dorian

Perfect, thanks a lot for the fast reply. Totally makes sense that it doesn't work in that case and I naively assumed the study ID to be just empty for the evaluation. Guess that should be easily resolvable and in case I'd stumble upon other errors I'll get back to your offer.

Thanks and best regards,

Johannes

@dorian_kauffmann

Currently, for the evaluation phase, the full content of the metrics file is available on the results detail page and via the API.

As the logs will only be seen by the organizers, I allow this to give you a better understanding of your results in the final submission.

Thanks, this perfectly answered my question.

If you don't think this is relevant, I can launch a poll asking all participants to display only the final scores publicly, or the entire content.

This doesn't change much in my opinion, but I would prefer results to be public when the submission is closed, or only for team/people who have already submitted.

Also :

I will see on Monday to make a poll about the relevance of giving one more week for the evaluation phase.

Is this survey going to take place?

@cyril.meyer.68

This doesn't change much in my opinion, but I would prefer results to be public when the submission is closed, or only for team/people who have already submitted.

Everything will be public :)

Is this survey going to take place?

If we have no submissions by this Thursday, March 13, we will consider adding a few extra days for all participants. However, if at least one participant has made a submission by this Thursday to be on schedule, we will not extend the deadline in the interests of fairness.

If we have no submissions by this Thursday, March 13, we will consider adding a few extra days for all participants. However, if at least one participant has made a submission by this Thursday to be on schedule, we will not extend the deadline in the interests of fairness.

jstegmaier just make a submission and mine is executing.

Good luck to all !