Expected Model Output

Hello,

Thanks for organizing. I have been trying to find a baseline algorithm so that I could check out exactly what output is expected, but I have not been able to find it. So I guess I have two questions:

1) Is there somewhere a baseline docker algorithm we can use as template to write our submissions?

2) Are you expecting the PDAC likelihoods (patient-level scores for AUC computation) to be submitted separately from the detection maps, as a list of floats? Or do you extract them out of the detection maps by taking the max confidence? Or should/may we provide an alternative likelihood extraction function, different of taking the max?

Cheers, Adrian

Hi Adrian,

Happy to answer your questions:

1) Yes there is a baseline algorithm available, you can find it in this Github repository or test it directly from grand-challenge. In the repository, there are also instructions on how the algorithm was trained.

2) The patient-level PDAC likelihood scores should be outputted as a single float written to a JSON file, while the detection map should be an MHA file with the same dimensions as the input image. The patient-level PDAC likelihood should be derived from the detection map using any function you find suitable. For the baseline algorithm, we indeed took the maximum value of the detection map, but feel free to explore other options.

Hope this helps! Don't hesitate to reach out if you have any further questions.

Hello Natalia, thanks for the quick reply and nice repo, btw, loads of work there.

I only have one more question (for now). Is it mandatory to derive patient-level likelihoods manually, and from detection maps? Like, imagine I decide its better to derive them directly from softmax outputs with some sort of integration, I guess that would be acceptable right? Or, instead of a manual approach I could use a learned approach and train a small neural network or whatever to take the detection map (and maybe some other stuff) and return a PDAC patient score, would that be admissible?

Regards, Adrian

Hi Adrian,

Thank you for your question. We do require the patient-level likelihood score to be derived from the detection map. The function you use to derive it can be as simple (like taking the maximum) or as complex (for instance, a neural network) as you make it. That being said, we would like to point out that using a more "indirect" approach could weaken the interpretability of your algorithm and introduce some inconsistencies. For instance, it would be worth considering what would happen in a scenario where a detection map with non-zero candidate lesions is provided as input to this function but the final PDAC likelihood turns out to be close to zero (or vice versa). Would you then update the detection map? Just questions worth taking into account.