Opening up the evaluation code

Dear all,

It is my pleasure to announce that the evaluation code is now publicly released on the SynthRAD2025 GitHub page. This code will be used to evaluate the quality of the synthetic CTs to the ground-truth CTs based on:

• Image similarity (as measured by the mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and multiscale structural similarity score (MS-SSIM))
• Geometric accuracy based on the segmentations generated by TotalSegmentator (as measured by the DICE similarity and 95th percentile of the Hausdorff distance)
• Dose accuracy for photon an proton radiation plans (as measured by the mean absolute dose error (MAE), the 2%/2mm gamma pass rate index, and the relative dose-volume histogram (DVH) differences in the target and organs at risk

There are two separate releases of the evaluation code on the SynthRAD2025 GitHub page:

• The metrics repository contains the standalone functions to estimate the synthetic CT quality. For example, the image_metrics.py file contains the functions to estimate the similarity between a synthetic CT and CT. Likewise, the segmentation_metrics.py contains the functions to estimate the segmentation quality given a ground-truth set of segmentations and estimated segmentations. The dose_metrics.py contains the dose accuracy functions, but the dose plans are currently not available.

• The evaluation repository contains all the tools to build an automated evaluation pipeline to estimate all the metrics given your synthetic CTs. This includes running TotalSegmenator on-the-fly on your synthetic CTs. However, this requires the availability of segmentations of your ground-truth data (which can be computed yourself using TotalSegmentator). Moreover, please carefully the accompanying README file to prepare the input data and make it ready for evaluation.

With this code, you should be able to reproduce the metric values when you submit your algorithm to Grand-Challenge.

Best of luck in the rest of the challenge!

On behalf of the SynthRAD2025 organizers,

Maarten Terpstra

This is great resource, thanks. When you say "The evaluation repository contains all the tools to build an automated evaluation pipeline to estimate all the metrics given your synthetic CTs.", does this include plan calculation? I'm trying to understand from the repo and it does not appear to be the case, but I may be wrong.

Thanks for your time.

Hi José,

This code does not include the planning code, which will be published after the post-challenge phase, as described here. However, it does contain the metrics to evaluate the dosimetric accuracy given that a doseplan is available.

If there are any further questions, please don't hesitate to reach out.

Best, Maarten

Dear Maarten,

Can we compute the DoseMetrics on the training data? I can't find the dose_path. Thanks!

I am using a Python script from the SynthRAD challenge to automate this validation process. The script is designed to call an external matRad process to perform dose recalculation on both the planning CT and the synthetic CT, and then compute metrics like DVH differences and Gamma Index. I have run into a specific technical issue on my Windows 10/11 system. The Python script contains the following line to execute the core computation: subprocess.call(["/matrad/compute_metrics", dose_path, patient_id, rad_type, sct_path], ...) As you can see, the script attempts to run a program located at /matrad/compute_metrics. This appears to be a Linux-style absolute path, and as expected, it fails on my Windows machine with a FileNotFoundError. My question is: What is the correct way to execute this compute_metrics process on a standard Windows installation of matRad?

Dear Hao Yang and André Filipe Sousa Ferreira,

The evaluation functions are intended to be run inside a Docker container. Here, these functions should run without problem on Windows or Linux. However, the doseplans are not publicly available for the SynthRAD2025 challenge. Therefore, there is no way to run the dose recalculation on the training or validation data, nor is it meaningful to run the compute_metrics matRad function as part of a validation pipeline.

If there are any other questions, please don't hesitate to reach out.

Best, Maarten