Evaluation of algorithm docker but with different adaptors, without updating the docker in algorithm

Hi, Is there a way to evaluate an algorithm but with different adaptors, without updating the docker image. As far as i can understand, currently the uniqueness of submission is only linked with the unique id of the docker container, which restricts it to be evaluated only once per submission per leaderboard. However there can be 2 cases where this can be counterproductive:

• Successful submission with one set of adaptors, but testing with other set of adaptors is restricted, thus asking us to furnish new updated docker container. This unnecessarily increases the number of docker containers per algorithm image.

• If there is a submission failure of a docker due to evaluation docker issues corresponding to an adaptor (for eg WeightedKNN Error for 1-nn-weighted), execution of same docker for other adaptors (like Linear Probing) should be allowed for the same docker.

One idea to achieve this could be to generate uid based on algorithm signature and adaptor signature, thus making submissions possible for different adaptors but same algorithm. But if there are other methods currently available to achieve the same behaviour that i may not be aware of, it would be great to know about them too.

Hi @aditya.vartak, thanks for your message!

You're right, by default each submission is linked to a unique Docker image. However, we do support running the same algorithm with different adaptors without requiring you to resubmit a modified Docker. You can request a re-evaluation of your existing submission with additional adaptors by sending an email to the organizing team at support@unicorn-challenge.com. You can find detailed instructions in our evaluation GitHub repo here. Let us know if anything is unclear or if you need help!