Problems in submission

Hello,

I found an error in the in the inference.py provided. Specifically in the read_image() function. Seems that the metadata is not in tif.shaped_metadata attribute (or at least not for all images). I'm trying locally with a example image (image_90_nucleus_angle.tif) and seems that the image has the metadata inside the imagej_metadata attribute. This is a loop over all the attributes I made trying to find where was the metadata:

 --> Predict image_90_nucleus_angle.tif
andor_metadata is None
astrotiff_metadata is None
avs_metadata is None
eer_metadata is None
epics_metadata is None
fei_metadata is None
fluoview_metadata is None
gdal_metadata is None
gdal_structural_metadata is None
geotiff_metadata is None
imagej_metadata: 
{'ImageJ': '1.11a', 'images': 204, 'slices': 204, 'hyperstack': True, 'mode': 'grayscale', 'spacing': 1.0, 'physicalsizex': 0.19500001, 'physical_size_x': 0.19500001, 'physicalsizey': 0.19500001, 'physical_size_y': 0.19500001, 'physicalsizez': 1.0, 'physical_size_z': 1.0, 'xresolution': 0.19500001, 'yresolution': 0.19500001, 'zresolution': 1.0, 'unit': 'um', 'study': 1, 'channel': 'nucleus'}
indica_metadata is None
lsm_metadata is None
mdgel_metadata is None
metaseries_metadata is None
micromanager_metadata is None
nih_metadata is None
ome_metadata is None
philips_metadata is None
pilatus_metadata is None
scanimage_metadata is None
scn_metadata is None
sem_metadata is None
shaped_metadata is None
sis_metadata is None
stk_metadata is None
streak_metadata is None
tvips_metadata is None

The read_function can be changed as follows:

def read_image(location): # WARNING IMAGE DATA EN ZYX
    # Read the TIFF file and get the image and metadata
    with tifffile.TiffFile(location) as tif:
        image_data = tif.asarray() # Extract image data
        # metadata = tif.shaped_metadata  # Get the existing metadata in a DICT
        metadata = None
        for x in dir(tif):
            if "metadata" in x:
                if getattr(tif,x) is not None and 'physicalsizex' in getattr(tif,x).keys():
                    metadata = getattr(tif,x)
                    break         
    if metadata is None:
        raise ValueError(f"Metadata not found in image {location}")
    return image_data, metadata

Also you need to change the following lines:

PhysicalSizeX = metadata['PhysicalSizeX']
PhysicalSizeY = metadata['PhysicalSizeY']

Into these:

PhysicalSizeX = metadata['physicalsizex']
PhysicalSizeY = metadata['physicalsizey']

Cause seems that the attribute names are in lowercase.

I don't know also how critical is the metadata in the process of saving the image, but so everyone is aware of this error. Maybe I did something wrong at some point and that's why this was happening (maybe the organizers can shed a bit of light on this). Anyway, this message is for all who are trying to create the submission so you don't waste the same time as me ;)

Cheers and good luck,

Dani

Thank you for this detailed topic !

I took a closer look and noticed that we use the imwrite function from tifffile to write the train and test images. However, due to the Grand Challenge setup when loading test set, we couldn't use the imagej=True parameter, so we created the inference code accordingly. It turns out this affects how metadata is read according to train and test images—I'm just discovering this now thanks to your post!
With ìmagej= True` all upper case letter was transformed into lowercases...

I've written a new read_imagefunction that standardize metadata regardless of the imagej writer parameter. It will avoid the errors abour the upper and lower case letters, according to the used writer (i.e. with or without the imagej parameter.)
I hope this can help some of you.

def standardize_metadata(metadata : dict):
    key_map = {
        "spacing": ["spacing"],
        "PhysicalSizeX": ["PhysicalSizeX", "physicalsizex", "physical_size_x"],
        "PhysicalSizeY": ["PhysicalSizeY", "physicalsizey", "physical_size_y"],
        "PhysicalSizeZ": ["PhysicalSizeZ", "physicalsizez", "physical_size_z"],
        "unit": ["unit"],
        "axes": ["axes"],
        "channel": ["channel"],
        "shape": ["shape"],
        "study": ["study"],
    }

    # Normalize metadata by looking up possible keys
    standardized_metadata = {}
    for standard_key, possible_keys in key_map.items():
        for key in possible_keys:
            if key in metadata:
                standardized_metadata[standard_key] = metadata[key]
                break  # Stop once we find the first available key

    return standardized_metadata



def read_image(location): # WARNING IMAGE DATA EN ZYX
    import tifffile
    # Read the TIFF file and get the image and metadata
    with tifffile.TiffFile(location) as tif:

        image_data = tif.asarray() # Extract image array data

        if tif.shaped_metadata is not None:
            shp_metadata = tif.shaped_metadata[0]
            metadata = standardize_metadata(shp_metadata)

            return image_data, metadata
        else:
            if tif.imagej_metadata is not None:
                shape = list(image_data.shape)
                imgj_metadata = tif.imagej_metadata
                imgj_metadata['shape'] = shape
                metadata = standardize_metadata(imgj_metadata)

                return image_data, metadata

            else:
                metadata = tif.pages[0].tags['ImageDescription'].value
                print(f"error loading metadata: {metadata}, type of object : {type(metadata)}")

Thank you for the piece of code, it works like a charm!

By the way, I have just submited my algorithm but it doesn't appear in the leaderboard, is there any specific reason for that? As in the previous challenge you organized, it is useful to see how your method ranks as it may decide whether to invest time into creating the 4-paper submission to the conference or not...

Cheers,

Dani

There was a very little error in the ranking display reading (about finding the results according to the good path), thank you for pointing it out!

I've corrected it and everything's in order now.

For ranking (officially in the next phase, but the same principle is applied here), we perform a ranking for each metric according to their scores. Then, the final/global rank is the average of the relative ranks of the metrics (to avoid normalization problems between two different metrics).

Is this clear enough for you? Otherwise, please don't hesitate to ask.

Everything clear, thanks Dorian!!

Hi everyone,

I have a quick question on the metrics: the top-scoring leaderboard results I submitted are actually obtained by just returning the input image without any adjustments, so my fusion algorithm currently doesn't seem to be very efficient so far :-). Guess that's also the "approach" the other group on rank 1 used, as the results are identical. I was wondering how it could happen that there are non-zero scores obtained. Wouldn't it be expected that n_ssim = 0 if prediction_ssim == reference_ssim ?

Thanks in advance for any clarifications and best regards,

Johannes

Hello,

We're looking into the details because your predictions are not black images and different images, the fact that you have identical scores is very odd.

Concerning the non-zero vaues /the negative values means that the SSIM is worse than the SSIM between the angle and the fusion (i.e. between the input and the ground truth). In this case this is odd that we do not obtain 0 in the case of prediction_ssim == reference_ssim...

We're trying to understand this further.

Hello,

Actually I confirm that one of the submissions is just an identity mapping returning the input to check the baseline performance - and interestingly it's the best performing one :)

Bests,