Inquiry about Region Information Accuracy in Test Data

Dear SynthRAD_2023 Community, In the provided code https://github.com/SynthRAD2023/algorithm-template for the competition, we noticed that you have used the region.json file to help us retrieve the region information for the data. However, it seems that the region.json file provides a single region information for the entire data folder, rather than providing specific region information for each individual data item.

During the preliminary test for Task 2 - CBCT, we also observed that our performance metrics for the last three data items were significantly lower compared to the first three. It appears that this discrepancy might be due to the region information provided in the region.json file not matching the actual region configuration of the last three data items.

To ensure clarity, we would like to confirm whether in the final testing phase, will there be genuine region information provided for each individual data item, or will there be separate folders for each region configuration?

Best regards, Chuanpu Li

Dear Chaunpu,

In the Grand Challenge platform, the algorithm runs for each of the case, this means that the content of the Region interface (.json) changes dynamically for each case. After having double-checked the archives, I can assure you that the regions are correctly set for the six patients in the preliminary set, you should have 3 "Head and Neck" (so brain) and 3 "Pelvis" patients. The same will hold for the test case (we are running tests on this). Moreover, we have already had confirmation from some participants that the option is working correctly, so I need to ask you to double-check whether you are correctly using this variable in your algorithm.

Note that the philosophy behind the region variable is that you may use it in case you wish to deploy a model for the brain anatomy and a model for the pelvis anatomy, or you are using the variable as input of your model. Is this what you are trying to achieve?

If you have proof that the region is not correctly set in a specific case, please pinpoint me to the error or show me how you gathered this information. I would be happy to rectify the archive. Moreover, if you feel like having a private conversation to try to debug the issue, please reach out by mail.

Best Regards,

Matteo

Dear Chuanpu,

I've double-checked the accuracy of the region information on a per-case basis on the preliminary test set.

To clarify how the algorithm dockers are run on GC and how to use the test folder to handle multiple cases, I've added a small note in the algorithm-template repo https://github.com/SynthRAD2023/algorithm-template/blob/master/README.md#note

If you'd like to confirm the region information yourself, you could add a print statement in the docker and inspect the logs as mentioned here

Feel free to reach out if there's anything else.

Best, Suraj