Use evalutils to package your Algorithm

If you wish not to worry about how data is loaded and written, and you just want to get your Algorithm on the platform with as few lines of code as possible, then you may want to use evalutils as is outlined on this page. We do note, however, that this approach is deprecated as it uses legacy-supported features of the Grand-Challenge platform. The approach on the next page is generally preferred.

In this tutorial, we will build an Algorithm container for a U-Net that segments retinal blood vessels from the DRIVE Challenge. We used evalutils to create the Algorithm container for the DRIVE challenge, and our scripts can be found here: https://github.com/DIAGNijmegen/drive-vessels-unet/tree/master/VesselSegmentation

The image below shows the output of a very simple U-Net that segments vessels.

To start the process, let's clone the repository that contains the weights from a pre-trained model and the python scripts to run inference on a new fundus image.

$ git clone https://github.com/DIAGNijmegen/drive-vessels-unet.git 

Create base repository using evalutils¶

Evalutils provides methods to wrap your algorithm in Docker containers. Just execute the following command in a terminal of your choice:

$ evalutils init algorithm VesselSegmentationContainer 

Here, VesselSegmentationContainer is the custom name we have given for our algorithm container for this walkthrough. This will create a templated repository with a Dockerfile.

The scripts for your container files will be automatically generated by evalutils. It also creates commands for building, testing, and exporting the algorithm container.

VesselSegmentationContainer/ 
├── Dockerfile 
├── README.md 
├── build.sh 
├── export.sh 
├── process.py 
├── requirements.txt 
├── test 
│   ├── 1.0.000.000000.0.00.0.0000000000.0000.0000000000.000.mhd 
│   ├── 1.0.000.000000.0.00.0.0000000000.0000.0000000000.000.zraw 
│   └── expected_output.json 
└── test.sh 

Bring your own Algorithm in process.py¶

The next step is to edit process.py. This is the file where you will extend the Algorithm class of evalutils and implement your inference algorithm. In this file, a new class, VesselSegmentationContainer, has been created for you, and it is instantiated and run with:

if __name__ == "__main__": 
  VesselSegmentationContainer().process() 

You can build on top of this template and write your algorithm into the repository. The default algorithm generated by evalutils does simple binary thresholding of the input image. You'll have to replace that with your custom algorithm.

For example, you can see how we have edited process.py for the DRIVE Algorithm based on inference.py in this version of process.py.

Configuring the Dockerfile¶

Ensure that you import the right base image in your Dockerfile. For our U-Net, we will build our Docker with the official PyTorch Docker as the base image. This should take care of installing PyTorch with the necessary CUDA environments inside your Docker. If you're using TensorFlow, please build your Docker with the official base image from TensorFlow. You can browse through Docker Hub to find your preferred base image. The base image can be specified in the first line of your Dockerfile:

FROM pytorch/pytorch 

Here are some best practices for configuring your Dockerfile.

🔩** Copying your model weights into the Docker **

Ensure that you copy all the files needed to run your scripts, including the model weights, into /opt/algorithm. This can be configured in the Dockerfile using the COPY command. If your model weights are stored in a "checkpoints" folder, first copy them into the repository which contains the Dockerfile. Then add this line into the Dockerfile to ensure the checkpoints are copied into the container when the Docker gets built.

COPY --chown=algorithm:algorithm checkpoints/ /opt/algorithm/checkpoints/ 

📝Configuring requirements.txt

Ensure that all of the dependencies with their versions are specified in requirements.txt as shown in the example below:

evalutils==0.3.0
monai==0.4.0
scikit-learn==1.0
scipy==1.6.3
scikit-image==0.18.1

Note that we haven't included torch as it comes with the PyTorch base image included in our Dockerfile in the previous step.