Configure Scoring¶

Under the Scoring tab, you can configure the leaderboard. This is where you connect the metrics generated by your evaluation container with the automated leaderboard mechanism present in grand-challenge.org.

Deadline: 6 weeks post-challenge acceptance

Single Metric¶

Assuming that your evaluation container writes the following scores as the output in /output/metrics.json:

{
    "malignancy_risk_auc": 0.85
}

You can configure the Scoring mechanisms as shown in the figure below.

• Make sure to give an appropriate Score title - this will be displayed as the column name in the leaderboard.
• Specify the JSON path of the main metric in Score jsonpath (in this case it is malignancy_risk_auc).

Multiple Metrics¶

You can also configure a more complex scoring mechanism. Take the following metrics.json, which is a nested dictionary:

{
  "case": {},
  "aggregates": [
    "dice": {
      "mean": 0.6,
      "std": 0.089
    },    
    "accuracy": {
      "mean": 0.5,
      "std": 0.00235
    }   
  ]
}

To use both dice and accuracy metrics in the scoring mechanism, and display both scores with their error on the leaderboard you would need to:

• Enter aggregates.dice.mean in Score jsonpath.
• Enter aggregates.dice.std in Score error jsonpath.
• Use Extra results columns to add the accuracy score and error to the leaderboard
• Set Scoring method choice to determine the way the scoring is calculated.

Additional Settings¶

Two additional settings under Scoring allow you to configure the runtime environment for your evaluation container:

• GPU type
• Memory limit

These influence which virtual machine instance is used to run the evaluation and should match the needs of your evaluation method. If the available options don’t cover your requirements, contact support.

Note that these settings only apply to evaluation containers. The runtime settings for participants' algorithm submissions are managed separately and are set together with the support team when setting up your challenge.

Under the Leaderboard section, you can configure the appearance of the leaderboard. Refer to the field descriptions for guidance.

Under Result Details, you can choose whether to display the complete output file from your evaluation container or just a summary of the metrics used for scoring in a clean, readable format.