predictions.json for evaluate.py

Dear Orgasination team,

I have tested the baseline and can successfully use the inference. However, in order to evaluate the trained models, I need the ‘prediction.json’ for evaluate.py, which I cannot find in the given code. Is there a script to generate the prediction.json, or should I create it myself? If so, what should it contain for the script to work properly?

Best regards

Dear Kabadood,

Thank you for your participation.

The evaluate.py script comes from the public luna25-evaluation-public repository (https://github.com/DIAGNijmegen/luna25-evaluation-public/tree/main). This repository includes the scripts that Grand-Challenge uses to process and evaluate the outputs of the submitted algorithms in order to calculate the performance metrics. The public repo does contain a prediction.json in ./test/inputs, which is there for testing purposes.

However, for your submission this is not something you have to configure yourself as this is done on the Grand-Challenge side.

If you need further help, feel free to reach out.

Kind regards,
Dre

Dear Drepeeters,

does this mean that the “evaluate.py” is only executed by the Grand Challenge team for our model and not by ourselves?

Best regards

Dear Kabadood,

The evaluate.py is included in the evaluation Docker, which is what we as organisers provide for evaluation. The evaluation Docker is executed after a participant submits their algorithm Docker, so you are not required to do this yourself.

Kind regards,
Dre Peeters

I see, thank you for the quick reply.

Best regards, Kabadood