Inquiry on Availability of Evaluation Script

Hello,

I've reviewed the metrics on the challenge website (Metrics & Ranking) and appreciate the clear guidelines provided. To better prepare our submissions and ensure alignment with the evaluation criteria, I was wondering if it would be possible for the organizers to release an official evaluation script. This would greatly assist in evaluating our models on local datasets before making submissions.

Thank you for considering this request and for all the efforts in organizing this challenge.

Hello pl552,

thanks for the suggestion, we now make our evaluation script publicly available on the official GitHub repository: https://github.com/cgsaxner/DREAMING-challenge/tree/main/evaluation

Hopefully, this helps prepare your submission!

Hi gsaxner,

Thank you very much for releasing the evaluation script. This is incredibly helpful, and your responsiveness is highly appreciated.

I have one additional question regarding the evaluation: Could you please provide more details on how the submissions will be ranked based on the evaluation criteria (Consistency/Plausibility, Accuracy, and Runtime)? Specifically, I'm interested in understanding if there is a particular order of priority among these criteria, or if the two errors (E_consistency and E_accuracy) are weighted equally in the ranking process. Besides, I would like to know when runtime will be considered and how it factors into the overall evaluation.

Thank you once again for your assistance! Best regards,

Hello pl552, you are welcome!

We will rank the methods according to consistency first, and accuracy second. This is because for Diminished Reality, a consistent result is more important than a pixel-accurate one. More information can also be found in the challenge design on Zenodo, which unfortunately only now became available: https://zenodo.org/records/10687606

Concerning the runtime, due to the internal architecture of grand-challenge.org, we decided not to include it in the official ranking. If possible, a separate ranking of algorithm runtimes will be presented at the challenge event.

Thank you for pointing these aspects out! We will add them to the documentation.