Try out an algorithm

Request Access¶

Prior to using an algorithm, you need to request access. You do not need to request permission if you are using your own algorithm. To request access, navigate to the algorithm, and click on "Request Access", as shown below. This request needs to be accepted by the Algorithm Editors before you can use the algorithm.

If the Algorithm Editors do not respond, the Grand Challenge support cannot bypass this access-request step. If this happens, you can contact the Algorithm Editors directly through their contact email listed on the algorithm page.

If the Algorithm requests are configured to only automatically approve requests of users that are verified, the pending request will be automatically approved when your account becomes verified.

Try-out Algorithm¶

To try out your own algorithm, you will have to wait until your algorithm container is uploaded and . You can check the status of your container by clicking on Containers. Normally, it takes about 20 minutes for your new container to be validated by grand-challenge.org and to become active. The easiest way to start a job is by navigating and clicking on the Try-out Algorithm option from the side menu or from the Results page

On this page, you can simply drag and drop your inputs or images, and then click on the Save button to start the job.

Connect an Algorithm to an Archive¶

Another way to run an algorithm job is to connect an Algorithm to an Archive. Go to your Archive → Update Settings → select an Algorithm.

Using gcapi¶

The third way to run algorithm jobs is to use the Grand Challenge API Client to start an algorithm job through Python.

Each method will require you to provide all the inputs defined for the algorithm. After submitting the inputs, the algorithm job will be queued and the job will be visible on the Results page.

Viewing results¶

To view the results of your jobs, click on the Results page and then click on your result to view the outputs, or click Open Result in Viewer to view your image with one of our image viewers.

When you click you Open Result in Viewer, you will be redirected to a CIRRUS viewer that will display the outputs of the algorithm like in this example from the CORADS-AI algorithm.

Logs and metrics¶

To view the logs of your algorithm, click on the Results page and then click on your result and View result details. In the tab Logs, you will find a chart, showing metrics regarding CPU, GPU and memory usage. It will also show stdout and stderr produced by your algorithm.

Credits¶

💳 Trying out an algorithm requires credits. The amount of credits per job will be automatically set depending on your algorithms compute requirements and historical runtime, with a minimum of 20 credits. A user has 1000 credits per month at their disposal.

Job permissions¶

The permissions for who can view each job depend on how the job was created. Each algorithm has its own editors group, and each job has its own viewers group. Users with the "Change this job" permission can add users to the job viewers group, granting them the "View this job" permission. Users with the "View this jobs logs" permission also need the "View this job" permission bit to actually view the logs.

Permission when the job is directly created by a user through the API or HTML Forms¶

Permission when the job is created as part of a challenge submission¶

Each challenge will have its own admin group who will be able to view the jobs and logs for debugging. The submission creator will not be able to see the job to prevent leaking of the test set.

Permission when the job is created as part of linking to an archive¶

Users with the "Use this algorithm" permission are able to link an algorithm to an archive so that it is executed on each new Archive Item. Each archive will have its own editors and users group who will be able to view the jobs. The algorithm editors will not be able to see the jobs to prevent access to any private data.

Job execution times¶

⚠️ The algorithm jobs are executed on AWS SageMaker where we limit the number of instances we use. This means that the job will only be executed when an instance becomes available. There is no way to predict how long it will take before an instance becomes available (might take more than a day) or to move a job further up in the queue.

Job status meaning¶

When you start an algorithm job it progresses through a number of different states. First, the job inputs are provisioned and the task is then put into the "provisioned" state. The task then waits for an available instance in our cluster. When your job is assigned to an instance it will be in the "executing" state. Once the algorithm job has run, your job's status will update to either "failed", "successful" or "cancelled".