Automated Evaluation


Every challenge has a unique way to objectively evaluate incoming submissions. More often than not, the evaluation scripts come with a set of dependencies and computational environments that are difficult to replicate in the host server. Therefore, we have decided that every challenge organizer has to provide a Docker container that packages the evaluation scripts. This container will run on our servers to compute the evaluation scripts necessary for an incoming submission.

Building your evaluation container

To make the process easier, we created evalutils. Evalutils is a Python package that helps you create a project structure, load and validate submissions, and packages the evaluation scripts in a Docker container compatible with grand-challenge.org. Note that you do not have to use evalutils.

Requirements

You can use your favorite Python environment to install evalutils.

pip install evalutils


Once you've installed the above requirements, you can follow the instructions for getting started and building your evaluation container here and watch the video below to see James take you through an example.


Uploading your evaluation container

Evalutils also provides methods to build, test and export your container. Watch the video below to get a walkthrough of these capabilities from James.


Once you have created your evaluation container, you can upload it to your challenge by selecting Admin → Methods → + Add a new method.