Inquiry About IMAGE_MASK_MAP and Time Constraints

Hi there,

I'm currently working on integrating my algorithm and I got 2 questions.

1. Our algorithm jobs are currently subject to a 10-minute time limit per case. Based on this, my understanding is that during the testing phase, the total completion time would amount to 20 minutes for two cases, and in the final phase, 100 minutes for ten cases. Is this interpretation accurate? Additionally, I was also wondering if there's any flexibility regarding these time limits. Is there any possibility of extending this time?

2. The IMAGE_MASK_MAP variable is hardcoded in helper.py in example_algorithm. Could this lead to a KeyError during actual submissions, and should it be modified to prevent such errors?

IMAGE_MASK_MAP = { "6c0da39e-2117-4da4-94dd-ba5e2f8188b7": "dcda170d-a3c0-4fd2-af4a-becc2722b3dd", "75904266-551a-4985-85d0-8d414151780b": "44c40c43-f866-443b-a848-647e950aebfe", }

Thank you in advance for your help!

Hi!

Concerning your first question: I don't have an option currently to extend the algorithm runtime, but I have reached out to the grand-challenge support team and will follow up with more information soon.

Concerning your second question: You are right, the values are hard-coded. Unfortunately, right now this is the only way to associate images and masks as they are stored with unique IDs in the grand-challenge.org Archives. I have now updated the IMAGE_MASK_MAP in the GitHub example_algorithm with the 10 final test cases:

helper.py

Please use this map in your final submission!

Thank you, Christina

Hi!

Update regarding Question 1: Each item in the DREAMING challenge Archive (so each test case) will spawn an algorithm job that will have a runtime limit of 10 minutes. As such, they are not cumulative: if an algorithm spends 1 minute on a case and tries to spend 15 minutes on another, it will be terminated at the 10 minutes mark.

We have selected the 10-minute limit based on our experiments with various inpainting algorithms, including our own designs. Grand-challenge can increase this 10-minute limit per case for an entire challenge phase. For now, we will not increase the runtime limit, because

a) As challenge organizers, we have to keep the challenge budget in mind since we have to pay for the GPU times participants' algorithms require.

b) We emphasize that runtimes should be kept short, as DR algorithms should eventually run in real-time (although this is not a requirement for participation).

If you absolutely require an extension, please reach out to us again with the approximate runtime you would need per case.

Thank you!

Hi Christina!

Thank you for your prompt update; it was indeed enlightening.

My situation is, I have reviewed the info webpage and the PDF provided before, but I was unaware of this time constraint until trying to submit.

Understanding the reasons behind the constraint, I fully respect the guidelines set forth. I have accelerated my model a lot. However, I will still kindly request if there might be any flexibility with the time limit. An extension, even if it were to merely double the time limit, if possible, would be very helpful.

This challenge has deeply engaged my interest, and I've committed substantial time to it, keen on completing my submission. I truly appreciate your consideration of my situation and the opportunity to partake.

Thank you once again, and I look forward to any possibility of the extension.

Warm regards, Pengze

Hi Pengze!

I completely understand your situation, and we are sorry that the time constraints were not more evident from the guidelines. It is a tricky trade-off for us to balance resources for our participants (runtime, possible submissions) and challenge budget, and of course, we are trying to meet our participants' requirements. It is our first challenge and we are learning as we go, and I appreciate your feedback a lot :)

I've seen that you have successfully submitted an algorithm now. Do you still need the extension of the runtime?

Best regards, Christina

Hi Christina,

Thank you for your understanding; that’s very kind of you.

Despite my earlier submission, I think we are still in need of the extension. That submission was to test the submission pipeline in case of other unforeseen issues. With the extension, I would be able to submit the full network, ensuring the integrity of the work.

This is also my first challenge, and your efforts are indeed helping me a lot to manage it. Thank you for the prompt guidance every time!

Warm regards, Pengze

Hi Pengze,

thanks for all your feedback! We have now increased the runtime limit for algorithms from 10 to 20 minutes. I hope this works for you.

Looking forward to your submission!