March 2025 Cycle Report

Published 24 March 2025

Parametrisation of evaluations¶

Challenge admins have a lot of flexibility in how to assess and evaluate their participants submissions. Sometimes, however, it would be useful to be able to further fine-tune the evaluation with participant-provided options. This cycle, we worked on making this parametrization of evaluations possible.

Challenge admins can now define additional inputs for each of their phases. When defined, they appear as inputs on the submission form together with the usual Algorithm dropdown. The participant-provided input value is then passed along to the evaluation method together with all the algorithm's outputs, and hence can be used to fine-tune the evaluation process.

The UNICORN challenge makes use of this new feature. UNICORN participants will be able to choose which aggregation function the evaluation method should use to process their algorithm outputs. The submission form for a UNICORN phase will then look as follows:

The chosen aggregation function will also appear on the leaderboard:

Finally, challenge admins can choose to re-evaluate a submission with a different aggregation function, without having to rerun the entire algorithm.

If you would like to configure additional inputs for your challenge phases, please get in touch with the support team.

Use Existing Answers for Ground Truth¶

Educational reader studies allow instant feedback and ranking of reader performance. For this, the reader studies rely on having a ground truth to compare the answers against.

Creating ground truth for an educational reader study is awkward, editors need to download or construe a CSV file, link the case identifier with the correct image they uploaded, and provide the ground truth in strange formats including single and double quotes. Multiple-choice answers make this even more awkward.

To streamline this process, we've introduced the option to create ground truth from existing answers. This means that an editor can first go through a reader study, answering the different questions, and with the press of a button, the answers can then be remade into ground truth.

Furthermore, we've added confirmation for the deletion and added checks for completeness of ground truth. A warning will let an editor know if the current ground truth does not cover all the cases or questions.

Invoice overview¶

Following the introduction of post-challenge invoices (see the December cycle report), we have added an overview of invoices. Similar to the onboarding task list, challenge organizers can now easily track the payment status of invoices. We have also automated email alerts when invoices are due, to ensure you are reminded as soon as possible and to avoid potential delays. The overview can be found in the challenge admin section.

Region growing for mask questions¶

We have added support for region growing in the mask question editor. It allows users to select one or more seed points in the image and set an absolute or relative threshold to perform region growing. The resulting region will be updated instantly when the settings or seed points are changed. When the user is satisfied with the result they can merge it with the existing manually created mask. For 3D images, a user can choose between 2D slice-based region growing or 3D region growing. See the following video for a demo of the feature.

Cover photo by TOMOKO UJI on Unsplash