Troubleshooting failed submissions

I've uploaded an algorithm which seems to function correctly when using "Try-out algorithm" with 007.tif from the github repo, i.e. it produces the correct json for the image. However, when I submit my algorithm to be ran on the preliminary test set, the evaluation is Failed The algorithm failed on one or more cases.. As far as I can tell there is no way to check the error output, so any help is appreciated.

I can't say for sure either (because I don't have access to the full log), but I can tell you that your first image produced a "core dumped" error.

Maybe the grand-challenge crew (or James in particular) will read this tomorrow and look up the trace to the error, if not, please try to contact grand-challenge support.

Meanwhile, you might want to try out with one of the training images. Note: I'm just uploading a new version of the training images to the google drive share (subfolder images_with_resolution) where the DPI information is set (the rest is totally equal). This is a requirement by the GC platform, so you might want to try out using these images.

Hope it helps,

best,

Marc

We do not release the logs to users as this could leak information about the test set, but the challenge administrators might be able to help if there is enough information in the logs. Unfortunately this particular algorithm container does not produce many logs, these are the relevant last lines:

Processing an image: 99%|█████████▊| 76/77 [02:09<00:01, 1.66s/it]free(): invalid next size (normal) 2021-08-23T17:16:10.441067149Z Aborted (core dumped)

Marc - I've sent you an email showing you how to reach the algorithm logs for each participant.

Thanks, James! The screencast was really helpful. :-) This means I also try to help better now - I just wasn't aware of the logs menu entry. Sometimes you just don't see the obvious!

Best,

Marc

Thanks for such prompt and helpful replies Marc and James! It turns out that I didn't look closely enough at the provided get_batch function in detection.py and had assumed that the function always returned a batch of size self.batchsize. This was true in 007.tiff, where the image happened to be evenly divided, but is not true in other images.

Also from James' message it seems like the python stack traces aren't going to the logs for some reason (maybe?), as I did get a fairly useful stack trace when reproducing this while running the docker container locally.

Huh, could you maybe send support the output of docker logs container_id when you run the same docker container image and input image as this job? We do run things differently in production and I would like to track down if it is a problem with our logging agent.