Clarification on Evaluation Setup for Task 2 and LUNA25

Dear Organizers,

Thank you for putting together such a comprehensive and well-structured evaluation challenge.

I have a question regarding the potential overlap between the UNICORN Task 2 (lung nodule malignancy classification in CT) and the ongoing LUNA25 challenge (https://luna25.grand-challenge.org/), which appears to target a very similar objective.

Both datasets seem to originate from NLST, but I noticed a discrepancy in dataset sizes: UNICORN includes 16,077 nodules (1,249 malignant, 14,828 benign), while LUNA25 uses 6,163 nodules (555 malignant, 5,608 benign).

Given this similarity, I would like to ask whether the evaluation setup for both challenges is aligned. Specifically, will the input structure for UNICORN Task 2 follow the same format as LUNA25, i.e.:

• Nodule Locations (multiple points): /input/nodule-locations.json
• Chest CT (image): /input/images/chest-ct/.mha or .tif
• Clinical Information (lung CT): /input/clinical-information-lung-ct.json

If the setup is indeed similar, I would consider reusing the same Docker container for both tasks.

Thank you in advance for your clarification!

Best regards,

Sebastian O.

Hi Sebastian,

Thanks for your message! Indeed UNICORN is for the most part, a collection of (mostly previous) challenges. For example Task 7 is based on LUNA16. While the objective in those challenges was to develop the best model on a single task, the objective of UNICORN is to develop the best feature extractor / adaptor combination on several tasks. In UNICORN you do not win prizes for the individual task leaderboard - it is not your objective to make the best malignancy predictor, but to make the best feature extractor that can be used (via an adaptor) for downstream lung nodule malignancy risk prediction and lung nodule detection and prostate cancer detection, etc.

That said, please do see the UNICORN challenge set up as completely unique. While LUNA25 follows a normal challenge design; large training data set provided which you must use, training occurs off-platform, only inference occurs on-platform, UNICORN requires the "training" to happen on-platform with a small amount of hidden data.

For task 2: you get 64 few shots, these are nodule blocks with the nodule in the centre, and you will get a larger amount of benchmarking data; these are also nodule blocks. We do not release the exact ratio of malignant : benign for our hidden few shots or benchmarking data on platform. 16,077 nodules refers to the total NLST dataset that we annotated, from which I extracted this 64 shot subset for the public few shots (on Zenodo). To clarify: On Zenodo right now you have the public few shots to give you an impression of what the shots will be like on platform. These are not the exact shots that will be on platform. The shots on the platform are hidden for the validation and test phase.

In UNICORN: You submit an algorithm docker to a particular leaderboard with your feature extractor which will extract features from the hidden few shots and the hidden benchmarking data, this happens on the platform. You either choose an existing one or submit your adaptor method, and we (the organizers) submit a docker that trains your adaptor with the features from the few shots and runs inference on the benchmarking data.

The metric for LUNA25 and Task 2 is the same; AUC, but the datasets for the validation phase and testing phase are different.

Does this answer your questions? We also have a recording of the webinar which explains the challenge design. Happy to answer any more questions about this :)

Thank you for your answer! All clear now :)