Cannot see logs (stdout and stderr) when submission failed

Hi,

I submitted my algorithm for the generation track : it failed but I cannot find the logs containing the errors. However I tested it on my local machine and also on the "Try Out Algorithm" tab before submitting, it showed "succeded with warnings" as I expected. Is there a way to access the logs for real submissions when it failed ? I guess I forgot to deal with a precise case which does not appear in the provided data.

Thanks, best regards, Guillaume

Dear Guillaume,

The container was killed because it exceeded the memory limit of 11g. The image it failed on has a dimension of 400x1024x1024.

Grand-challenge do not release the logs to users as this could leak information about the test set (which is private), but we, as the challenge administrators, might be able to help if there is enough information in the logs. So please keep writing in the forum when you have any problem, and we will help you with the logs as soon as we can :)

Best, Ecem

Thanks for your reactivity and your help :) I guess I missunderstand something. As I read on the baseline GitHub : "For the sake of time efficiency in the evaluation process of NODE21, the submitted algorithms to NODE21 are expected to operate on a 3D image which consists of multiple CXR images stacked together. The algorithm should go through the slices (CXR images) one by one and process them individually, as shown in predict." So the input is a 3D volume of shape Nx1024x1024 where N is the number of CXR to process, and the output has exactly the same size, right ? In this case, it means that the container processes a volume of shape 400x1024x1024, which seems logical if 400 CXR have to be processed. Do I miss something or do I simply need to increase the memory limit in order to allow huge 3D volumes to be processed ? In the second case, what is the maximum N (number of CXR to process) for experimental test set and final test set ?

Thanks, best regards, Guillaume

Hi Guillaume,

You understand it correctly, indeed your algorithm will process 400 chest x-rays as a single image with dimension 400x1024x1024 and will produce an image with the same dimension. So, you indeed simply increase the memory in grand-challenge when you upload your container. The maximum dimension your algorithm will process for experimental and final test set is 400x1024x1024. Can you upload it again and increase this to a higher memory requirements?

I updated the memory limit to 24gb and submitted, let's see if it's enough.

Thanks again for your help ! Best regards, Guillaume

Hi Ecem,

My algorithm was running during 3 days, I thought it was working but just taking a lot of time, but it finally failed. Is it normal that it is so long ? Can you give me the reason why it failed please ?

Thanks a lot for your help, best regards, Guillaume

Hi Guillaume,

We cannot see the error, the output was not produced. Grand-challenge team is re-running your algorithm to investigate what went wrong, we will keep you updated.

Thanks, Ecem

Hi Ecem,

Ok thanks. To help them, I have maybe some clues : 1) I always get a warning during container building "debconf: delaying package configuration, since apt-utils is not installed". I think it's not necessary to install apt-utils but maybe I should. 2) I initially changed the labels in the Dockerfile (the 6 "nl.diagnijmegen.rse.algorithm.hardware"), especially I decreased info about cpu. I cannot find proper documentation about it, maybe I was wrong. I'm going to submit with a new container with original labels (same as on github), just in case. 3) When I execute process.py, I always get "Could not load expected_output.json using " and same for nodules.json , I don't know how to solve it. Maybe it causes an error during validation. You seemed to have the same issue as I can see on the youtube video you provided to help building docker container. I don't know if you uploaded exactly the same container as the one you built in the video (which causes the same warnings), so maybe this warning interrupts the validation script for some reason.

The test method from "Try out Algorithm" is also very long, around 8 hours for 5 images. Maybe it is related to my problem. Please tell me when you have some news from Grand-Challenge.

Best regards, Guillaume

Hi Ecem, after the changes I mentionned above, I submitted again and got 2 notifications : one saying that it succeeded then another one saying that it failed ("No such file or directory: '/input/predictions.json'") and a score appeared on the leaderboard. Score details look confusing as I get the same value (very low) for sensitivity_5, sensitivity_25 and sensitivity_125. Moreover it gave a result in less than 3 hours, while before I needed at least 24 hours. Do you think we can trust this submission ? Otherwise, is the trained detector the same as the detection baseline ? Thanks in advance, Best regards, Guillaume Sallé

Hi Guillaume,

The score details might be correct if the algorithm does not locate the nodules well; so even increasing the number of FP cases, you might not get increase in the detection sensitivity.

When you submit your algorithm, it is waiting to be assigned to a GPU node in grand-challenge, so thats why sometimes it takes more than a day to run your algorithm. If it runs in 3 hours, that just means that it got assigned to a GPU immediately.

Trained detector is the same with the detection baseline, so that you could also try to train it locally.

But, i will download the output of your algoritm (generated data), and will train the baseline model locally to make sure that there is no error there. One important thing to check to see if you are generating the nodules in the correct location (any mistake in x-y coordinate?) maybe you could take a look at the template code for the generation track and make sure that your generated nodules are in the correct location?

I will let you know when i finish the experiments with your data. Best, Ecem