Extension Final Phase

Dear finalists,

Thank you again for your efforts on producing your Final Phase submissions, and also for your patience with the training delay. All submitted training containers have now been run on the Final phase training set (9000+ cases). Unfortunately, most Final phase submissions did not successfully complete training (see footnote*). We recognize that this is likely due to the lack of sufficient feedback for most finalists to produce functional training containers .

We have therefore decided to extend the Final phase with extra opportunities for receiving feedback about your submission, followed by one more round of training on the Final phase training set. The details of this extension of the Final phase are described below.

Please note that making a second submission to the Final Phase is optional. If your first Final phase submission was successful, you will not be at a disadvantage since no methodological changes are allowed for the second Final Phase submission. If you make a second Final phase submission, your first Final phase submission will not be taken into account.

Details extension Final phase:¶

First a quick summary of the Final phase extension: If your submission failed, you will first receive the error that was generated by your Final phase training run. Subsequently, you get the opportunity to submit a codebase that the challenge organizers will run in the training environment for 24 hours. To provide you with the feedback you need to get your Final phase submission to complete successfully, you will receive any artifacts that this run generates. Finally, you get the opportunity to resolve any issues that prevented your training container from completing successfully, without making other updates to your methods. You will be able to submit your updated Final Phase codebase for one last training run on the Final phase training set. This will be your last submission, even if your submission fails within the first hour of running.

Below, you will find a more detailed explanation of the Final phase extension, as well as a timeline:

Resolving any errors generated in your Final Phase run.¶

Today, you will receive an e-mail with the details of your training run on the Final phase training set. If you do not receive this e-mail today, please respond to this announcement, or contact luuk.boulogne@radboudumc.nl.

Running in the training environment with access to the public training set.¶

To get more feedback, each finalist team can optionally submit a "feedback codebase". The challenge organizers will build and run this codebase in the training environment for 24 hours. For this run, the training environment will be identical to the environment used in the rest of the Final phase (see the Final phase baseline repository and the challenge overview), with the exception that you will have access to the public training set, instead of the Final phase training set. After your feedback run finishes, you will receive the logs and any artifacts that your container saved in the /output directory up to a maximum of 10GB. We recommend to base your feedback codebase on your original submission for the Final phase. However, you can make any changes to your codebase to obtain the feedback you need, as long as it adheres to the following requirements: * The GitHub repository used for your feedback codebase is different from the one used for your original Final Phase submission. This is required in order to clearly separate this submission and its commit history from your Final Phase submission. * Your feedback codebase follows the format of the Final phase submission template.

Note that your feedback codebase submission will not count towards your ranking on the Final Leaderboard in any way.

We propose a deadline for submitting your feedback codebase on July 17th at 23:59 in the Anywhere on Earth Time Zone. If this does not provide you with sufficient time, please let us know as soon as possible, but before July 13th, by responding to this announcement or by sending an e-mail to luuk.boulogne@radboudumc.nl with another deadline proposal.

You can submit your feedback codebase by sending an e-mail with the subject "STOIC2021: Feedback codebase" to luuk.boulogne@radboudumc.nl. This e-mail should contain: * Your team name, * A link to a GitHub repository with the feedback codebase that you want to submit. This must a different GitHub repository than your original Final phase submission. * A confirmation that you have added "luukboulogne" as a collaborator to this GitHub repository. * The git commit hash corresponds to your submission. Making a release is not necessary.

A second submission to the Final phase¶

After obtaining your feedback run logs and artifacts, you will get one week to optionally update your Final phase codebase. This update may not contain any additions to or deletions from your original submission, other than fixes of issues that prevented your training and/or inference container from running and completing successfully. Other updates to your methods are not allowed. To prevent that this last training run fails, we recommend to make sure that your training algorithm gracefully handles any errors that it generates. Also, please make sure to write artifacts (e.g. model checkpoints) to the /output directory every (few) epochs or after every epoch that resulted in a performance improvement.

You can make your second Final phase submission by first creating a new release for your Final phase codebase entitled "Final Phase 2nd Submission". You can complete your submission by sending an e-mail with the subject "STOIC2021: Final phase 2nd submission" to luuk.boulogne@radboudumc.nl. This e-mail must contain: * Your team name, * A link to a GitHub repository with the codebase that you want to submit. This should be the same link as your original Final phase submission. * The git commit hash that corresponding to the "Final Phase 2nd Submission" release. Note that the state of your GitHub repository at this release will be regarded as your second submission to the Final phase.

Note that if you make a second Final Phase submission, your first submission will not be taken into account. In this case, your second Final phase submission will be regarded as your only submission to the Final Phase.

Timeline:¶

We hope that all issues with compute instance availability have been resolved and that we will be able to adhere to the following timeline: * Deadline submitting feedback codebase: July 17th at 23:59 in the Anywhere on Earth Time Zone * Finalists receive the logs and artifacts generated by their feedback codebase: July 20th * Deadline second Final phase submission: July 27th at 23:59 in the Anywhere on Earth Time Zone

If you have any questions, please do not hesitate to contact me.

Kind regards, Luuk

*Overview of training runs: One submission failed in the first hour of training, meeting the original conditions detailed in the challenge overview for an extra training run. 3 out of 8 submissions successfully completed the training run.

The deadlines in this announcement have been updated.