Dear all,

It is my pleasure to announce that the evaluation code is now publicly released on the SynthRAD2025 GitHub page. This code will be used to evaluate the quality of the synthetic CTs to the ground-truth CTs based on:

  • Image similarity (as measured by the mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and multiscale structural similarity score (MS-SSIM))
  • Geometric accuracy based on the segmentations generated by TotalSegmentator (as measured by the DICE similarity and 95th percentile of the Hausdorff distance)
  • Dose accuracy for photon an proton radiation plans (as measured by the mean absolute dose error (MAE), the 2%/2mm gamma pass rate index, and the relative dose-volume histogram (DVH) differences in the target and organs at risk

There are two separate releases of the evaluation code on the SynthRAD2025 GitHub page:

  • The metrics repository contains the standalone functions to estimate the synthetic CT quality. For example, the image_metrics.py file contains the functions to estimate the similarity between a synthetic CT and CT. Likewise, the segmentation_metrics.py contains the functions to estimate the segmentation quality given a ground-truth set of segmentations and estimated segmentations. The dose_metrics.py contains the dose accuracy functions, but the dose plans are currently not available.

  • The evaluation repository contains all the tools to build an automated evaluation pipeline to estimate all the metrics given your synthetic CTs. This includes running TotalSegmenator on-the-fly on your synthetic CTs. However, this requires the availability of segmentations of your ground-truth data (which can be computed yourself using TotalSegmentator). Moreover, please carefully the accompanying README file to prepare the input data and make it ready for evaluation.

With this code, you should be able to reproduce the metric values when you submit your algorithm to Grand-Challenge.

Best of luck in the rest of the challenge!

On behalf of the SynthRAD2025 organizers,

Maarten Terpstra