Should we implement the img read function to input all images in the test folder?

From the baseline code, the get_image_file_path() function fixed to output the very 1st element in the test folder: https://github.com/DIAGNijmegen/ACOUSLIC-AI-baseline/blob/main/inference.py#L110. Not sure if we should change it to output the whole list or just the first one?

Thank you!

Q2: Another question I just post here together: I tried with my algorithm once, but it returns "Failed The algorithm failed on one or more cases." May I know if we can have more info regarding what is going on there? Thanks again!

Q3: I found that in the baseline code, https://github.com/DIAGNijmegen/ACOUSLIC-AI-baseline/tree/main/test/input/images/stacked-fetal-ultrasound, the given test sample is just one sweep. May I know if the online evaluation also adopts sweep-by-sweep testing or the stacked 6-sweep imgs as in the training samples?

Dear participant,

Thank you for your questions. I'll reply to all of them here:

Q1: Should we implement the img read function to input all images in the test folder?

The get_image_file_path() function manages data loading internally in the grand-challenge platform. Grand-challenge runs the algorithm separately one case at a time. The loading and saving functions should not be edited, as they are designed by the grand-challenge team to manage their internal system for handling job submissions. You should thus not edit this function to read all input images.

Q2: I tried with my algorithm once, but it returns "Failed The algorithm failed on one or more cases." May I know if we can have more info regarding what is going on there?

I will message you privately to provide more information for debugging your algorithm submission.

Q3: I found that in the baseline code, the given test sample is just one sweep. May I know if the online evaluation also adopts sweep-by-sweep testing or the stacked 6-sweep imgs as in the training samples?

The test sample included in the baseline code is limited to one sweep to avoid saving unnecessarily large files on GitHub. When you submit your algorithm, the inference is run in the whole stack of 6 sweep images. Your algorithm is expected to output a binary 2D segmentation mask and the corresponding frame number in the range [0, 840), and -1 whenever no good frame is found. Then, the evaluation code evaluates your solution at the image and sweep level:

• For the dice metric, it considers the mask provided and the corresponding one (indicated by the frame number) in the ground-truth, if present.
• For Hausdorff distance computation - which is not considered in the ranking metric - the closest frame within a range of 15 frames in the same sweep is considered to compute the 3D distance to the provided solution.
• For the circumference measurement, the mean circumference in the given sweep, if present, is considered.

Thank you, Sofia, for the detailed answers. They are very helpful. For Q3, may I double-check if the stacked 6 sweep images in a single file (the same in the training data), or they are individual 6 sweep files inputted to the model.

Thanks again!

You are welcome! They are 6 sweeps images stacked into the same single file per inference case :)