Live Leaderboard failed but Debugging Leaderboard successful?!

Hi, our algorithm works in the debugging leaderboard but fails in live leaderboard. Could you give us a hint regarding the error you see in the log file?

Additionally, we are a bit confused regarding the output format. It is not necessary to create a predictions.json file, correct? The points can still be in pixels or do they have to be converted to mm now?

Any help is highly appreciated. Thanks!

Best regards

Hi!

Are you referring to https://monkey.grand-challenge.org/evaluation/live-leaderboard/submissions/87df2640-f373-48c1-9544-dd04fad5b6fd/?

This is what I see in the log: cv2.error: OpenCV(4.10.0) /io/opencv/modules/core/src/alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 80044818432 bytes in function 'OutOfMemoryError'

The available resources should be the same as those used for debugging and live leaderboarding.

The output JSON files must be in mm (this has always been the case). You don't need to create the predictions.json. GC automatically generates that. I only included it in the GitHub code for the evaluation to show how it runs on the backend. You can check out the inference tutorial about how to save the JSON files.

Hi,

thanks for the reply.

Yes, that is our submission. Thanks for the log message. This helps!

Best regards

I see that you had another failed submission. The error log is TypeError: 'Polygon' object is not iterable. I suggest you submit to the debugging phase to validate your docker before submitting to the live leaderboard phase :)! Will make it easier for you, as you can see the logs there.

Hi Linda, thanks for your reply. The method passes all local and debugging tests, it only fails on the live leaderboard check. Is there anything special about the test set, like much larger images? Otherwise, I can't explain why it passes all other tests.

Hi! No, it should all be the same. Your latest submission has again the error TypeError: 'Polygon' object is not iterable. Are you sure you are submitting the correct container? The error sounds like something that is happening in the code. I've not had any complaints from anybody else so far, and I see there were other successful submissions yesterday.

Hi! Yes, we now also encountered this issue locally on a slide we haven't tested on earlier. So at least this error should be fixed now. :) Thanks!

This is happening in our case as well . can you help with the logs.

@prateek: I see Output file 'detected-lymphocytes.json' was not produced as an error for your latest submission.

Full error log:

2024-12-14T05:21:31.929000+00:00 |2024-12-14|05:21:24.074| [INFO] .... Detect #GPUS: 1
2024-12-14T05:21:31.930000+00:00 Traceback (most recent call last):
2024-12-14T05:21:31.930000+00:00   File "hovernet_mod/run_infer.py", line 182, in <module>
2024-12-14T05:21:31.930000+00:00     infer.process_file_list(run_args)
2024-12-14T05:21:31.930000+00:00   File "/app/hovernet_mod/infer/tile.py", line 162, in process_file_list
2024-12-14T05:21:31.930000+00:00     assert len(file_path_list) > 0, 'Not Detected Any Files From Path'
2024-12-14T05:21:31.930000+00:00 AssertionError: Not Detected Any Files From Path
2024-12-14T05:21:31.930000+00:00 
2024-12-14T05:21:31.930000+00:00 0it [00:00, ?it/s]
2024-12-14T05:21:31.930000+00:00 0it [00:00, ?it/s]
2024-12-14T05:21:31.930000+00:00