Handle the processing of inputs and outputs yourself

Evalutils currently provides templates for only three generic tasks: classification, segmentation, and detection. These templates take care of loading data, and writing outputs in a way that is specific to the task: you are only required to adjust the predict function. If, however, the data or task does not fit the templates, you must also write code to load inputs and write outputs. This pretty much equates to writing the whole process.py script yourself.

Writing it yourself gets you fine-grained control over how inputs and outputs are read and written. But this also means that you have to choose input and output interfaces. Interfaces tell Grand Challenge what kind of data it has to deal with, and where to get or put it. Interfaces have to be selected on the Grand Challenge website, and may need to be requested.

The general principle is that your Algorithm container will process one job at a time, and each job will process only one set of inputs. So you need to write your scripts to read that one set of inputs from a location in /input and write your algorithm's outputs to a location in  /output. For the default algorithms, evalutils automatically does this in the background. But for algorithms that require flexibility, you will have to write your own code. Check the snippets below for an example.

First of all, we run evalutils from the command line to generate a template, and call it CustomAlgorithm (but you can pick a more suitable name, of course):

$ evalutils init algorithm CustomAlgorithm

In the example below, we choose the segmentation template. You can choose whichever template fits your use case the best, but it does not matter as much, in this case, because we won't inherit the base class anymore. The automatically generated code for CustomAlgorithm  would look like the following:

import SimpleITK
import numpy as np

from evalutils import SegmentationAlgorithm
from evalutils.validators import (
    UniquePathIndicesValidator,
    UniqueImagesValidator,
)

class Customalgorithm(SegmentationAlgorithm):
    def __init__(self):
        super().__init__(
            validators=dict(
                input_image=(
                    UniqueImagesValidator(),
                    UniquePathIndicesValidator(),
                )
            ),
        )

    def predict(self, *, input_image: SimpleITK.Image) -> SimpleITK.Image:
        # Segment all values greater than 2 in the input image
        return SimpleITK.BinaryThreshold(
            image1=input_image, lowerThreshold=2, insideValue=1, outsideValue=0
        )

if __name__ == "__main__":
    Customalgorithm().process()

To get more fine-grained control over inputs and outputs, we write our own process function. In addition to running inference, this function should take care of loading a set of inputs from /inputs/ and writing a set of outputs to /outputs/ . To this end, we choose to write a load_inputs and a write_outputs function in the example below, and call them in the process function. Also, note that we let go of inheriting the base class from the template:

import SimpleITK
import numpy as np

class Customalgorithm():  # SegmentationAlgorithm is not inherited in this class anymore
    def __init__(self):
        """
        Write your own input validators here
        Initialize your model etc.
        """
        pass

    def load_inputs(self):
        """
        Read from /input/
        Check https://grand-challenge.org/algorithms/interfaces/
        """
        return inputs

    def write_outputs(self, outputs):
        """
        Write to /output/
        Check https://grand-challenge.org/algorithms/interfaces/
        """
        pass
    
    def predict(self, inputs):
        """
        Your algorithm goes here
        """        
        return outputs

    def process(self):
        """
        Read inputs from /input, process with your algorithm and write to /output
        """
        inputs = self.load_inputs()
        outputs = self.predict(inputs)
        self.write_outputs(outputs)

if __name__ == "__main__":
    Customalgorithm().process()

Adding files and dependencies¶

When you're adding files to the algorithm folder, such as model weights, you have to edit the Dockerfile by adding the following line for every additional file:

COPY --chown=algorithm:algorithm checkpoint /opt/algorithm/checkpoint 

Replace both instances of  checkpoint with the file path that you need for your Algorithm to run. In addition, any additional dependencies will need to be added to the  requirements.txt file with the version number specified, which looks like this:

evalutils==0.3.0 
scikit-learn==1.0 
scipy==1.6.3 
scikit-image==0.18.1