Change in the reference docker container

Dear all,

please be aware of a recent change in the docker reference container (today, July 25). Compared to MIDOG 2021, we changed the output format of the containers to not only include above-threshold detections but also below-threshold detections. This was a requirement in order to calculate threshold-independent metrics (like average precision).

If you have any question in this regard, please contact us.

Best regards,

Marc

Hi Marc,

My docker image builds just fine. But during test i get this problem: Could not load expected_output.json using . I checked out the github issue: https://github.com/comic/evalutils/issues/310 but there is no solution there but the problem is described. I am able to generate bounding boxes in the correct format when printing out:

[{'point': [0.7115890383720398, 0.7606987953186035, 0], 'probability': 0.6003159284591675, 'name': 'mitotic figure'}, {'point': [0.6594460606575012, 0.5152987241744995, 0], 'probability': 0.6070718765258789, 'name': 'mitotic figure'}, {'point': [0.6113260388374329, 0.5173661112785339, 0], 'probability': 0.6095658540725708, 'name': 'mitotic figure'}, {'point': [0.5222033262252808, 0.5135414600372314, 0], 'probability': 0.6732180118560791, 'name': 'mitotic figure'}]

My test file runs the docker commands perfectly for the gpu run but the next two docker runs for the output json give me the above mentioned error: Could not load expected_output.json using . Due to this my tar.gz docker container cannot be used or submitted. Kindly help me out with this.

Dear Ishan,

during which part of the test do you get this issue?

Which environment are you in? Windows? Linux?

Do you use the most recent version of evalutils?

Also: Did you modify any of the files that are related to building the docker container itself?

Best,

Marc

Hi Marc,

I'm running the algorithm on linux.

These two docker commands fail, namely:

docker run --rm \ -v mitosisdetection-output:/output/ \ python:3.7-slim cat /output/mitotic-figures.json | python -m json.tool

docker run --rm \ -v mitosisdetection-output:/output/ \ -v $SCRIPTPATH/test/:/input/images/histopathology-roi-cropout/ \ python:3.7-slim python -c "import json, sys; f1 = json.load(open('/output/mitotic-figures.json')); f2 = json.load(open('/input/images/histopathology-roi-cropout/expected_output.json')); sys.exit(f1 != f2);"

If i comment these two parts my tests pass with the docker run --rm --gpus all, else i get the following errors:

1. Could not load expected_output.json using .
2. ./test.sh: line 17: python: command not found write /dev/stdout: broken pipe Expected output was not found...

Dear Ishan,

are you using the most recent version of the evalutils package?

Best,

Marc

So it appears that the container is reading the JSON file expected_output.json as an input image that it needs to process. This should not (or maybe no longer) be the case.

to my previous message, when i changed python to python3 in the following highlighted bold text below in the test.sh script:

docker run --rm \ -v mitosisdetection-output:/output/ \ python:3.7-slim cat /output/mitotic-figures.json | python3 -m json.tool

docker run --rm \ -v mitosisdetection-output:/output/ \ -v $SCRIPTPATH/test/:/input/images/histopathology-roi-cropout/ \ python:3.7-slim python3 -c "import json, sys; f1 = json.load(open('/output/mitotic-figures.json')); f2 = json.load(open('/input/images/histopathology-roi-cropout/expected_output.json')); sys.exit(f1 != f2);"

i can see it printing successfully:

{ "type": "Multiple points", "points": [ { "point": [ 0.7115890383720398, 0.7606987953186035, 0 ], "probability": 0.6003159284591675, "name": "mitotic figure" }] }

but the following error persists:

1. Could not load expected_output.json using .
2. Expected output was not found... - this i am guessing is to be expected as my model prediction will not be the same as your MIDOG model prediction

I will try submitting again and report my findings

Latest evalutils is: 0.3.1

I was using what was given in the MIDOG reference docker which is: evalutils==0.2.4

So, I checked again with our reference container. Here, we also get a warning WARNING:evalutils.evalutils:Could not load expected_output.json using .

This is related to the test trying to run on all files of the test-folder, which includes the expected_output.json.

But this is merely a warning, it is still possible to run the test fine.

There's several ways of getting rid of this warning:

• Adding a file_filters-Argument to the constructor of your Detection class, which prohibits loading the JSON file.
• Moving of the expected_output.json from the test folder. Of course this will lead to a fail of the equality check in the test.sh script, which would likely fail anyway (since you'll not be providing the same output very likely anyways).

Best,

Marc

I tried resubmitting after making the changes, it gives the Image Can be Used Flag as False. I get the output printed on the terminal with the detection points properly but still there is a problem with the json file.

Just to Recap:

during which part of the test do you get this issue?

docker run --rm \ -v mitosisdetection-output:/output/ \ python:3.7-slim cat /output/mitotic-figures.json | python -m json.tool

docker run --rm \ -v mitosisdetection-output:/output/ \ -v $SCRIPTPATH/test/:/input/images/histopathology-roi-cropout/ \ python:3.7-slim python -c "import json, sys; f1 = json.load(open('/output/mitotic-figures.json')); f2 = json.load(open('/input/images/histopathology-roi-cropout/expected_output.json')); sys.exit(f1 != f2);"

I had to change 'python' to 'python3' for running the commands mentioned in the test.sh script so that it generates output. But it still gives me the error: Could not load expected_output.json using Hence, i finally get Image Can Be Used: False, when i upload the image

Which environment are you in? Windows? Linux? Linux

Do you use the most recent version of evalutils? evalutils==0.3.1 also tried evalutils==0.2.4

Also: Did you modify any of the files that are related to building the docker container itself? Didn't modify any files that didn't have the TODO Flag

I'm not able to figure out what the problem is from my side as the code runs fine and generates the output, but it seems as you mentioned earlier that: the container is reading the JSON file expected_output.json as an input image which it shouldn't do and should ideally ignore.

Is there someway to debug the above issue and find out where the real problem is ? Kindly let me know.

Thanks Marc

• Ishan

Dear Ishan,

did you try to delete the expected_output.json from the test folder?

As on my system the message you wrote only triggers a WARNING not an error, I wonder if there might be another error that is actually the problem.

Best,

Marc

Hi Marc,

Please find my answers inline:

did you try to delete the expected_output.json from the test folder? I tried this and it showed more errors. didn't solve the problem

Second thing you asked me to do: I Initialized the constructor in the detection class with file_filters but didn't know what to pass into it. It was very difficult to understand the eval utils file and it's parameters to block the json file.

But surprisingly my docker container image has been passed. I think it took a lot of time to pass through as it was around 7.4 GB.

Thank you for all your help. This was my first every grand challenge and it would not have been possible without your help. Thanks again :-)

• Ishan