Question about the Hausdorff distance

Hello,

I have a question about the HD used to evaluate the algorithms. As far I understand the direct Hausdorff distance is used for the evaluation (not the 95th percentile). So far, results from all participants in the Phase 1 seem to be quite poor in this aspect, thus I have the following questions:

1) What is the unit of the HD in the leaderboard? Is it in mm or in voxels? 2) Since even a single wrongly predicted / annotated voxel may strongly influence the HD - isn't there any problem with the associated annotations? I have quickly checked the provided training cases and for some of them there are outliers that influence the HD (but still not at the level as in the hidden test set).

I am asking because I became a bit suspicious about the obtained results. Error at the level of ~180 mm is quite a lot for the aorta segmentation and is basically equal to significant portion of the input volume itself.

Bests,

We were analysing this detail also on our side. We currently do not have access to the single predictions, but according to the Dice score they look promising.

It is possible that the high values in the HD are due to the different test set (different clinical center) and the actual number of detected branches. This is part of the challenge goals.

To reduce the effect of these variables, we have now switched the evaluation to HD95 in millimeters. All teams are allowed one extra test run (up to 6) to allow everybody to compute the updated HD value.

Best, Antonio

Thanks for your response.

Initially I was assuming the same - that the problem may be connected with different clinical center and the number of detected branches. However, I would suspect the HD at the level of maximum a few cm. The 18 cm seems a bit to large.

By the way - I created another submission and the 95th percentile of the HD is absolutely equal to the old HD. I think there is a bug.

Thanks for your message.

It should be fine now.

We were also able to identify which portion of the image is creating the issue. We are now excluding this portion, so all participants should be both a reduction in HD95 and a small increase in DC.

We hope that all teams will be happy with this change -- We thought that it is fine to make this change now as we are not in Phase 2 yet.

Best Antonio

Thanks, will try to resubmit the container tomorrow.

By the way - how should we interpret the "null" value in the Jacobian of the surface mesh after the conversion into volumetric mesh? I mean - is it an indication that there is any division by 0 or infinite values in the computation after the volumetric mesh is calculated?

Bests,

Hi,

The issue is most likely due to the mesh not being watertight. I have added a check which should now prevent null values.

Best, Antonio

Thanks for the reply. Now all the meshes (for the volumetric meshing) should be watertight. Nevertheless, as posted in another thread - the system still throws an error.

Bests,