PSNR and SSIM are not good metrics, missmatched HEs and IHCs may be the reason.

Here are two predictions (exp1 and exp2) of test set: Link: https://pan.baidu.com/s/1K10dNQryWYAFv8OEMawFsg Password: 00p1

exp1: average PSNR: 22.6520 average SSIM: 0.5576

exp2: average PSNR: 23.8398 average SSIM: 0.6184

exp2's metrics are better than exp1's. However, it can be observed that predictions of exp2 are worse than exp1's by visually comparing. It is the same on val set.

PSNR and SSIM give us some missleading to evaluate predictions. How do we deal with this issue?

Thank you very much for bringing this issue to our attention！We consider inviting doctors to evaluate the results submitted by the participants. If the generated images in a submission do not interfere with the doctors' observation and diagnosis, it will be considered as a valid submission. Participants should refrain from submitting results of significantly low quality (e.g. cell structure is completely destroyed).

There are many missmatched pairs of HE and IHC in train, val and test dataset.

For example: train set: 00009_train_2+ 00032_train_3+ 00109_train_3+ 00255_train_2+ 00271_train_3+ 00272_train_3+

val set: 00029_train_2+ 00605_train_2+ 00644_train_1+ 00917_train_2+ 01073_train_2+

test set: 00009_test_2+ 00013_test_2+ 00034_test_2+ 00038_test_2+ 00088_test_1+

Above examples are part of missmatched pairs, are these normal?

Missmatched HEs and IHCs are similar in image structure but not in cell structure. This may be the reason that PSNR and SSIM are not consistent with good performance.

The preparation process of the pathological slides resulted in the image mismatches. For a piece of pathological tissue, the doctor will cut two tissue samples from it for HE staining and HER2 detection. Therefore, there will be differences in the morphology of the two pathological samples. Besides, the tissue samples will be stretched or squeezed to a certain extent during slice preparation, which will increase the difference between the samples. However, there is currently no technology that can well achieve the registration of ultra-high-resolution pathological images. The HE-IHC image pairs we processed also have partially misalignments, which will also be one of the problems that the pathological image translation algorithms need to deal with.