Validation image padding

Dear organizers of the DENTEX challenge,

When running the Docker container, it seems that the 50 validation images are given as a 50xHxW array, where the H and W are presumably the maximum height and width across all validation images.

I was wondering how exactly smaller images are padded, such that they can be concatenated to the largest image. We would like to unpad each validation image, to allow the model the process the smaller images at a higher resolution.

Could you elaborate on how exactly the padding is done? Are you simply adding rows and columns of zeros?

Kind regards,

Niels van Nistelrooij

Dear Niels van Nistelrooij,

Thank you very much for your inquiry. We simply add columns and rows of zeros so that you can easily apply unpadding with the provided image height and widths in the tables. In the DENTEX repository, you can find a list of dictionaries containing the widths and heights, along with the file names and image IDs in the process.py file within the docker image codes. Below is the function we used to add the paddings:

def create_mha_from_rgb_images(images, output_path):
    # Find the dimensions of the largest image
    max_height = max(img.shape[0] for img in images)
    max_width = max(img.shape[1] for img in images)

    # Create an empty 3D array to hold the RGB images
    stacked_image = np.zeros((max_height, max_width, len(images), 3), dtype=np.uint8)

    # Copy each RGB image into the corresponding layer of the 3D array
    for i, img in enumerate(images):
        # Zero-pad the image if necessary
        padded_img = np.pad(img, ((0, max_height - img.shape[0]), (0, max_width - img.shape[1]), (0, 0)), mode='constant')

        # Assign the padded image to the appropriate layer
        stacked_image[:, :, i, :] = padded_img

    # Convert the array to SimpleITK image format
    sitk_image = sitk.GetImageFromArray(stacked_image)

    # Set the pixel type and origin

    sitk_image.SetOrigin((0.0, 0.0, 0.0))

    # Save the image as .mha file
    sitk.WriteImage(sitk_image, output_path)

Ideally, the model should not predict bounding boxes (or segmentation masks in your case) in the zero-padding regions. Therefore, unpadding may not be necessary. In fact, we did not perform unpadding in the baseline model docker, and it appears to work fine.

Sincerely, Sezgin

Dear organizers,

Thank you for the clarifications!

I have uploaded my Docker container and tested it with the Try-out Algorithm function.

However, when submitting the Algorithm to the challenge, I am hitting a time limit of Grand Challenge, as the algorithm took 42 minutes to process 50 images, whereas Grand Challenge has a time limit of 20 minutes.

Could you change the setup of the submissions, such that the algorithms are run on a single image at a time? Otherwise, I am certain our algorithm will never pass the time limit when processing the 250 test images in one session.

Thank you for your consideration!

Kind regards,

Niels van Nistelrooij

Dear Niels van Nistelrooij,

Thank you for bringing to our attention the issue regarding the extended processing time of your model. Unfortunately, evaluating images individually significantly increases the GPU cost by approximately 5-6 times compared to evaluating them in the batch of 50. As far as I understood from the logs, you are loading 50 images one by one to your model for the inference. I wanted to ask if it would be feasible to modify your model for batch inference. If your model supports handling 50 images within the designated time limit, we could potentially divide the last phase into 50-image batches.

Best regards, Sezgin

Dear Sezgin,

We managed to shorten the processing time to 16 minutes, but the Docker submission still failed.

Could you provide the logs, such that we can understand where our algorithm is failing?

Best,

Niels

Dear Niels van Nistelrooij, I am having a same issue and my submission failed with the error 'failed on one or more cases'. How did you solve this issue? Thank you.

Dear EvidentAI,

This error could be caused by a number of things. To help you debug, I would suggest:

1. When testing the Docker container locally, set --shm-size=64m, as Grand Challenge uses the default amount of shared memory;
2. Furthermore, allocate VRAM at the start of your process.py to have 16GB available during processing. I have a GPU with 48 GB, so I use torch.zeros(8 * 1024 ** 3, device='cuda'). Obviously, remove this for the GC submission;
3. Make the single MHA file of all validation images yourself and test the Docker container with it locally. I can share this file, if you are interested.
4. Once the container works locally with the single MHA file, upload the image and try it with the Try-Out Algorithm feature. Note the processing time from the utilization graph;
5. If it is less than 20 minutes, submit the image to the challenge;
6. If it still fails, ask the organizers for the logs. If the log suddenly stops, you hit the time limit and otherwise the log includes a run-time error;
7. Thus, use good logging to keep track of where your algorithm fails when testing locally and on GC.

I hope this helps and good luck with your submission!

Best,

Shank

Dear EvidentAI,

Your algorithm's docker image apparently did not start. Can you make sure you created the algorithm image properly as shown our example GitHub. I also can provide further help if you can provide me more context if you can provide me your algorithm image code by e-mail.

Best regards, Sezgin