Bug in froc calculation

Dear organisers,

I think there is a bug in the current code for froc calculation, specifically here:

for i,thresh in enumerate(thresholds):
    for gt, pred in zip(gt_coords, pred_coords):
        if len(pred)==0:
            fns[i]+=len(gt) #no tp or fp
        elif len(gt)==0:
            fps[i]+=len(pred) #no fn or tp
        else:
            thresh_pred = pred[np.where(pred[:,2]>=thresh)[0],:2]
            det_score = score_detection(ground_truth=gt, predictions=thresh_pred, radius=dist_thresh)
            tps[i]+=det_score.true_positives
            fns[i]+=det_score.false_negatives
            fps[i]+=det_score.false_positives

For each threshold, we wish to calculate the false positives for that threshold.

When no true cells are present in the the image (len(gt) == 0), then the false positives should not equal the number of all predictions.

They should equal the number of predictions above the threshold in question. So the line below should be changed:

https://github.com/DIAGNijmegen/pathology-tiger-algorithm-example/blob/884678d8c4556e3b28788d6cef6ff497deecd427/evaluations/eval_det_froc_snippet.py#L28

Does this make sense or am I missing something?

Thank you Anna Maria

Hi Anna

I agree with your opinion. Current FROC algorithm heavily counts FPs on the image with no GTs. BTW, your code snippet link is broken, and I think the code on the link has not been fixed. Code snippet should be changed as below.

for i,thresh in enumerate(thresholds):

    for gt, pred in zip(gt_coords, pred_coords):
            if len(pred)==0:
                    fns[i]+=len(gt) #no tp or fp
            elif len(gt)==0:
                         thresh_pred = pred[np.where(pred[:,2]>=thresh)[0],:2]
                    fps[i]+=len(thres_pred) #no fn or tp
                    else
                        ...

Dear organizers, I hope we can discuss this issue in the second webinar.

Dear Anna Maria and Kyungdoc Kim,

Yes we also agree and have fixed this now. Please see this announcement

Thank you again Anna Maria for pointing us to this issue!

Best wishes, Mart