Documentation

How can I limit my compute costs?ΒΆ

There are numerous ways for you to control your compute costs. Measures you can take include:

  • Limiting the number of participants or teams
    If you enable "manual participant review", you decide who gets to submit solutions to your challenge, and if you have reached a certain number, you can stop accepting people to your challenge. Letting participants team up can also reduce the number of submissions.

  • Limiting the number of submissions
    The number of submissions participants can be limited in total or during a specified submission window.
    Have a public debugging phase with a small number of cases for testing rather than a large number. You can then allow for example only a single submission on the test set.

  • Putting a reasonable upper limit on algorithm job run times.
    We enforce this limit on the single job level, i.e., for the processing of a single set of inputs. Regardless of the costs, limiting algorithm run times is desirable since truly clinically useful algorithms will benefit from being fast, so forcing your participants to develop efficient solutions is a good thing to do.
    On the other hand, this constrains the complexity of the solution and if it's too short it will not be possible to create a useful algorithm.

  • Have a 2 stage design
    Only invite a subset of the participants to submit on the full test set rather than everyone.

  • Reduce the number of cases in the test set
    However, this will affect the statistical power of the challenge.

  • Batching the input data.
    If your test data set consists of a large number of very small images, you might be better off batching your inputs. The reason for this is that GC starts one algorithm job per input image (i.e., archive item), so the more images you have, the more jobs need to be started which increases costs. The downside to this approach is that the resulting algorithms will not be directly useful for clinicians, who will usually want to process a single (unbatched) image input. The integrated web viewer on Grand Challenge is also not equipped to read and display batched images, and hence algorithm result viewing will not be possible with such a design.