Spring newsletter 2023
Published 25 May 2023
🛠️ New features of Grand Challenge¶
Spring is in the air, and so are the latest updates from Grand Challenge. As we welcome the season of renewal, we're excited to share the latest Grand Challenge news with you.
Our team has been working hard to improve user experience. We're thrilled to announce that we've revamped the documentation on submitting your algorithm to Grand Challenge. With our new step-by-step approach, it's now easier to navigate the submission process and get your algorithm up and running quickly. And don't worry if you have any questions or need further guidance. Our team is always here to help!
There have also been some minor changes in the challenge admin page that should make it a easier to edit phase settings, upload evaluation containers and keep track of submissions to each of your challenge phases. For participants of challenges, we have furthermore introduced a limit to the number of algorithms they can create. This was done to encourage and make them aware of the fact that they can reuse one and the same algorithm for subsequent submissions to a challenge by simply uploading a new container. To provide even more flexibility, we have also added the option to reactivate old container images for algorithms - sparing your participants valuable upload time.
And, as the cherry on top, we're gearing up for the much-anticipated MICCAI challenges - stay tuned on the challenge page for more information on these upcoming events. As always, we're committed to providing the best experience for our community, and we hope these updates will help make your Grand Challenge journey even more rewarding.
In the shoes of challenge organizers!
¶Recently, we interviewed various challenge organizers to learn more about their experience with hosting challenges on Grand Challenge and what their main takeaways were to fellow future challenge organizers. Read it here!
What made you choose Grand Challenge over other challenge platforms?¶
For most organizers, it was the reputation in the field of biomedical imaging and our extensive user base that made Grand Challenge stand out. Organizers also valued the relatively low cost involved in challenge administration compared to other platforms offering similarly flexible and customizable automated evaluation and leaderboards.
What went well in the set-up and the actual execution of the challenge? Which features did you appreciate having access to?¶
The organizers highlighted several positive aspects regarding the setup and execution of the challenge. Access to high compute resources, such as GPUs, and the docker container submission feature were much appreciated, as well as the highly customizable live leaderboard functionality. Overall, organizers also valued the continuous, prompt support from the Grand Challenge Support team in setting up the challenge effectively.
Did you run into any issues/obstacles with your challenge set-up or during the hosting of the challenge later on? If so, what kind of issues and how (if at all) were you able to resolve them?¶
Organizers stressed a few important aspects that future organizers should consider before setting up their challenge. When your desired input or output (image) formats differ from file formats supported by Grand Challenge, take into account the time needed to adjust to workable formats for Grand Challenge. Challenge organizers should also know that setting up the evaluation of Docker container submissions usually takes some time and test submissions to get right and requires agreeing on a feasible set-up with the Grand Challenge support team (for more information, look here). Luckily, all of these things can be mitigated if you start early with the set-up and get in touch with the support team about possible solutions as soon as your challenge has been accepted. Next to you starting early, our support team is also working on improving the documentation for the evaluation container set-up. Stay tuned on updates.
If you would do it all over again, what would you do differently? Do you have any general advice for future challenge organizers?¶
A few nice takeaways were suggested by the organizers: develop a thorough study protocol, seek input from peers, reach out to experts and data centers globally to foster collaboration, and preregister the final protocol to enhance scientific credibility. This also includes carefully considering and clearly specifying the rules of the challenge in advance, based on examples from other challenges, by addressing common issues such as co-authorship, pre-training, and manual annotations. Examining the forum sections of other challenges can help anticipate participant questions and proactively address them. Moreover, it was advised to set a final submission deadline and avoid extending it unless necessary due to technical issues to maintain fairness. Equally as important, starting early was exhaustively addressed, especially when aiming to align the challenge with leading conferences, as data annotation and testing phases may require significant time. Lastly, allotting more time for testing phases, given the potential need for debugging with individual teams, was recommended.
Feel free to provide your own feedback by contacting us!
💡 Blogposts¶
"March 2023 Cycle Report"¶
Read all about the platform improvements from our RSE team in March, including an upgrade of the algorithm-job permissions and a new reader-study workflow.
"April 2023 Cycle Report"¶
In the April cycle, the RSE team added new features for reader studies, for which metadata can now be annotated for example, and added more advanced algorithm container management options. Read more about it in this blog post!
🔦 Highlighted algorithms¶
Breast and fibroglandular tissue segmentation
This algorithm produces a volumetric segmentation mask for the fatty tissue and the fibroglandular tissue in the breast given as input either a fat (FS) or without fat suppression (WOFS) T1 breast MR acquisition. The output of this algorithm allows us to easily calculate breast density from an FS or WOFS T1 breast MR acquisition. Try out this algorithm with your own data!
Prostate Segmentation
This algorithm segments the whole prostate gland in biparametric MRI (bpMRI). The development of this model was geared toward robust prostate segmentation at the expense of fine-grained zonal segmentation. The dice similarity coefficient is 0.8968 ± 0.0547 performance with a Jaccard index of 0.8169 ± 0 across the cases. Try out this algorithm with your own data!🚀 Upcoming and running challenges¶
Grand Challenge currently offers two types of submissions: prediction submission and algorithm container submission. The algorithm container submission type has the advantage of producing reproducible algorithms that remain accessible to the research community long after the challenge has ended. This allows for continued use and exploration of the algorithms by the community. Therefore, it should be noted that we are phasing out the prediction submission procedure in favour of the algorithm submission procedure to ensure that challenges always produce reproducible algorithms. To learn more about hosting a challenge on our platform, go to our documentation!
🥅 Goal: This challenge consists of 2 tasks:
1. Accurate detection and segmentation of FDG-avid tumor lesions in whole body FDG-PET/CT
2. Robust behavior of the algorithms in terms of moderate changes in acquisition protocol or acquisition site
✍️ Register: Training set released on March 1st, 2023
⏰ Deadline: Deadline final algorithm submission on September 22th, 2023
🏆 Prize: This challenge has 3 award categories:
Award Category 1: Highest overall score
1. €2.000,-
2. €1.500,-
3. €1.000,-
4. €500,-
Award Category 2: Highest robustness across domains
1. €1.500,-
2. €1.000,-
3. €500,-
Award Category 3: Jury prizes for scientific contribution and creative engineering
Scientific contribution: €1.000,-
Engineering contribution: €2.000,-
🥅 Goal: An AI challenge on providing a comprehensive benchmark for cephalometric landmark detection methods
✍️ Register: Training set released on May 1st, 2023
⏰ Deadline: Deadline final algorithm submission on August 15th, 2023
🏆 Prize:
- The top 3 performing methods will be awarded a certificate and €500
- A challenge paper will be written with the organizing team’s members in submission to journals. Top-performing teams will be invited to contribute to a challenge overview paper, which will be submitted to a high-impact journal (MedIA/TMI)
🥅 Goal: An AI challenge on dental enumeration and diagnosis on panoramic X-rays
✍️ Register: Training set released on April 10th, 2023
⏰ Deadline: Deadline final algorithm submission on August 15th, 2023
🏆 Prize:
Unknown
🥅 Goal: An AI challenge on developing automatic segmented mediastinal lymph nodes in CT scans
✍️ Register: Training set released on May 5th, 2023
⏰ Deadline: Deadline final algorithm submission on July 30th, 2023
🏆 Prize:
1. $1.000,- cash
2. $500,- cash
3. $500,- cash
🥅 Goal: An AI challenge on cell detection from cell-tissue interaction
✍️ Register: Training set released on April 10th, 2023
⏰ Deadline: Validation and test dataset release on August 11th, 2023
🏆 Prize:
1. $1.000,-
2. $500,-
3. $200,-
🥅 Goal: An AI challenge on aortic vessel tree segmentation
✍️ Register: Training set released onMay 15th, 2023
⏰ Deadline: Final submissions on August 15th, 2023
🏆 Prize: Unknown
🥅 Goal: This challenge consists of 2 tasks:
1. Segmentation of 45 organs at risk from no-contrast and contrast-enhanced CT scans
2. Segmentation of Gross Target Volume of nasopharynx and Gross Target Volume of lymph node from no-contrast and contrast-enhanced CT scans
✍️ Register: Training set released on April 22nd, 2023
⏰ Deadline: Final submissions on September 10th, 2023
🏆 Prize:
1. $1.000,-
2. $600,-
3. $400,-
🥅 Goal: This challenge consists of 2 tasks:
1. Generate synthesized CT from MRI to facilitate MR-only radiotherapy
2. Generate synthesized CT from cone-beam CT to facilitate image-guided adaptive radiation therapy
✍️ Register: Accepting submissions from April 1st, 2023
⏰ Deadline: The deadline for registering method paper on August 15th, 2023
🏆 Prize:
Depending on the available funding, organizers reserve the possibility to award prizes to the top teams in both tasks. To date, prizes amount to a total of 5000 euros per task
🥅 Goal: An AI challenge on segmentation of the inferior alveolar nerve
✍️ Register: Accepting submissions from March 30th, 2023
⏰ Deadline: Submission date on July 31st, 2023
🏆 Prize:
Prizes will be awarded by the challenge sponsors and will be defined soon
🥅 Goal: An AI challenge on Circle of Willis angio-architecture extraction and brain vessel segment annotation on two imaging modalities: magnetic resonance angiography (MRA) and computed tomography angiography (CTA)
✍️ Register: Training data release on June 30th, 2023
⏰ Deadline: Closing of submission system on September 15th, 2023
🏆 Prize:
Unknown
🏆 Leaderboard of finished challenges¶
The task for this challenge was two-fold: computation of tumor-cellularity in pancreatic cancer with tumor cell nucleus segmentation and calculating tumor-cellularity in colon cancer with tumor cell nucleus segmentation. The overall winner was Joyce Fang. Congratulations!
