Grand Challenge

Dear participants,

we would like to share some further information on how you can submit your solutions for the final stage of Learn2Reg 2023.

The submission is open from today, 14/09/2023, until 21/09/2023 and as we already mentioned in the last post, there are two options:

The preferred option is a submission as a GrandChallenge Algorithm and we encourage all participants to use this opportunity to get a bonus in our ranking as the submission of an algorithm allows for inference time evaluation.

We provide three baseline algorithms to simplify the process of integrating your own model into the required docker container. For the NLST task, we based our registration method on MONAI and you can find a tutorial here. For the ThoraxCBCT task we implemented Voxelmorph++. Additionally, we provide a Zerofield algorithm that can be used for both tasks.

Use one of our baseline algorithms to create your own algorithm by cloning the repository, replacing our baseline code with your own inference code in process.py and replacing the model weights. A detailed description of these steps can be found here. More information on the GC algorithms and how to build and test your docker container can be found here.

Our baseline algorithms are configured with four inputs and a single output (displacement field). The inputs are a fixed image and a moving image as well as a fixed and moving mask. In your own algorithm you are not required to use the masks but you have the possibility since the masks for both, NLST and ThoraxCBCT, are included in the test datasets. This is new for ThoraxCBCT, as we did not publish any masks for the training data. Unlike NLST, where lung masks are available, we decided to include body trunk masks for the ThoraxCBCT test data, as the lung is not the primary focus of the registration task. It is not possible to use more or different inputs or outputs for your algorithm.

Please note that SimpleITK is used to load the input images and masks and save the displacement field since GrandChallenge operates with .mha files instead of .nii. With SimpleITK the inputs are loaded in (D,W,H) format and therefore in our baselines are permuted into (H,W,D) format as they would result with nibabel and .nii inputs (see comments in process.py). Please do not perform any permutations when saving the displacement field with SimpleITK, our evaluation works with the (D,W,H) format!

To simplify creating your own algorithm we provide the possibility of a sanity check on Learn2Reg-Test. In the phases NLST Validation and ThoraxCBCT Validation you can submit your algorithm and it will be executed for the first fixed/moving image pair of the validation data of the task. We would like to note that the validation phase should be used to check the containerisation of your algorithm instead of validating your registration method.

Participants are allowed to submit their algorithms once to the test phase of the respective task and up to three times per day to the validation phase for sanity checks. We suggest contacting us in case your sanity check fails repeatedly. The memory limit for an algorithm is 16GB GPU memory (more information here) and there is a maximum runtime of five minutes per case. Please refer to Learn2Reg-Test for more information on how to submit your algorithm.

We encourage all participants to submit their solution as a GrandChallenge algorithm, as this allows for a fair runtime calculation in our ranking. In case of any questions on how to create and submit your algorithm, do not hesitate to contact us via email or the GC forum.
The second option to submit your solution is to calculate the displacement fields on the test data locally and upload them here. Important: Since we cannot obtain information about the inference time of your algorithm in this way, we will assign the measured highest running time of the GC algorithms to these submissions for statistical ranking. Different from the validation phase there won’t be a leaderboard on Learn2Reg, since the displacement fields are not submitted to GrandChallenge directly.
Although the focus of this year's learn2reg Challenge is on the ThoracicCBCT and NLST tasks, we invite all participants to submit to the AbdomenMRCT, OASIS and LungCT tasks as well. However, since we cannot compare inference times on these tasks with previous years, one can only submit displacement fields for this task. After filling out the questionnaire, participants will receive the download link to the test data, as well as further information about the submission.

We decided to not make our test data publicly available and instead request the participants to fill out the form we posted previously. We will make the test data available to you once we obtain your contact data from the form and ask you to use the data for the purpose of submitting to the Learn2Reg 2023 test phase only. The test data for the NLST task includes 15 pairs of images along with corresponding lung masks. As already mentioned previously we decided to publish body trunk masks for ThoraxCBCT that can be found along 10 test image pairs. For more information on the test data please refer to the dataset.json and info.txt that you can find along the data.

The evaluation and ranking of your solutions will be task-specific and based on the following metrics:

NLST:
- Landmark-TRE, 30th percentile of Landmark-TRE, standard deviation of logarithmized Jacobian determinant, lesion alignment, inference time.
ThoraxCBCT:
- Dice, 30th percentile of Dice, Landmark-TRE, 30th percentile of Landmark-TRE, Standard deviation of logarithmized Jacobian Determinant, inference time. The ranking will only be affected by relevant time differences (> 3 sec).
AbdomenMRCT, OASIS, LungCT:
- Same as previous years.

We would also like to remind all participants to fill out the form regarding their participation in the MICCAI L2R workshop: https://forms.gle/MW8Ss3tGJor8ChXR7 and of the opportunity to submit a paper to our special issue on image registration in the open-access MELBA journal until October 1: https://www.melba-journal.org/blog/012-special-issue-image-registration.html.

Please do not hesitate to contact us via email or here in the forum if you have any questions.

Best regards,

Alessa, Christoph, Mattias and Wiebke

Opening of Final Submission

Opening of Final Submission ¶

Re: Opening of Final Submission ¶