March 2023 Cycle Report

Published 20 March 2023

In this March cycle, the research-software engineer team has realized these new features:

  • More statistics on Grand Challenge
  • Updated algorithm-job permissions on Grand Challenge
  • Upgrade the CIRRUS backbone MeVisLab version to v3.6
  • A new reader-study workflow for accepting and rejecting findings in both CIRRUS and Grand Challenge

Grand Challenge

More Statistics

The Grand Challenge Statistics page now displays charts of data broken down by month, and each challenge also gets its own statistics page with a break down of the participants' locations and submissions.

Changes to Algorithm Job Permissions

Previously, algorithm editors would get view access to algorithm jobs for QA purposes. We have changed the job permissions so that algorithm editors will no longer have permission to do this for future jobs. If you wish to share your job for QA purposes please add the editor to the job viewers group. Please see the updated documentation for what permissions people get with different jobs on grand challenge.

To monitor usage, algorithm editors now get an overview of how often their algorithm is used and the top users for their algorithm:

CIRRUS viewer

MeVisLab 3.6 upgrade

We have updated CIRRUS internal MeVisLab version to MeVisLab 3.6. The updated MeVisLab supports Python 3.9 and contains general Fraunhofer MeVis updates that we might be able to use in the future. The new build tools used in the new MeVisLab version also allow for some cleanups in the CIRRUS codebase.

Accept or reject findings

In many algorithms there is a detection component, be it (multiple) points, bounding boxes, lines, angles, or polygons. In this cycle, we introduce a way to accept or reject findings from algorithms via reader studies:

Configuring a reader-study to use this new workflow requires two steps:

  1. Create a question that specifies the appropriate interface and select 'Accept / Reject Findings' as a widget.
  2. For each case or display set, add a new item with the same interface as the question in the previous step.

The CIRRUS viewer will automatically source the annotations to be reviewed from the correct display set. Once posted the answer contains only the accepted and additional findings, the rejected findings will have been filtered out.

Note that the widget will not allow the reader to progress unless each annotation is either accepted or rejected. When the question is additionally made to be required there is a minimum of one accepted/additional finding before the reader can progress through to the next case.

Calendar versioning

The astute reader might have already noticed that the versioning used for both Grand Challenge and CIRRUS viewer has switched from semantic to calendar versioning (e.g. 8.4.1 -> 2023.03.0). More specifically, we will be using YYYY.Cycle.Release from now on for all releases as we feel it better fits with how the team of research engineers operates.

Authors

Tags