Calculate evaluation metrics in inference.py?

Dear team,

Thank you for organizing this event. I just have two questions from my side -

1. Do we need to calculate the evaluation metrics in "inference.py" before uploading the docker container? If so, in the docker template that you shared, the "input.zip" does not have any ground truth file in the folder. Are we allowed to use any other input-output pair for evaluation purpsoe?
2. In the case that "algorithm failed on one or more cases", is there a possibility of viewing the log of the failed case?

Thank You, Gurucharan

Hi Gurucharan,

Do we need to calculate the evaluation metrics in "inference.py" before uploading the docker container?

No. Evaluation metrics are computed directly on our side. The data we shared is only the 'input', and your docker will create its corresponding 'output' when running 'test_run.sh'. Please stick to the provided github repo, as should lead to a working submission: https://github.com/ezequieldlrosa/isles24-docker-template

Are we allowed to use any other input-output pair for evaluation purposes?

Inputs and outputs are fixed. There is some other information on this discussion: https://grand-challenge.org/forums/forum/ischemic-stroke-lesion-segmentation-challenge-2024-722/topic/input-and-output-2626/ Note that in the git-repo above provided, all possible inputs are commented out; feel free to uncomment and load the data you want.

2) I just changed the configuration settings. For new algorithm submissions, you'll be able to see the full logs and fix the docker accordingly. In the meantime, i sent you by private message your current logs.

Best, Ezequiel

Thank you for the quick response, Ezequiel!

Best, Gurucharan