Create a Reader Study

Published 10 Feb. 2021

A unique feature of grand-challenge.org are reader studies. It enables users to create an online medical image viewer which can be used for different purposes. A reader study offers the possibility to upload medical images and add questions. They offer the following functionality:

  1. Ability to upload and host a set of medical images along with questions
  2. Invite participants to view the images and answer the questions
  3. Store and export results

There are two types of Reader Study:

  1. In a Regular Reader Study each participant is asked to answer questions about a number of medical images and the answers are recorded.

  2. An Educational Reader Study is meant as a teaching tool. It does everything a regular Reader Study does, and in addition gives the participant feedback on how they did on each question. An Educational Reader Study requires you to provide the correct answers for each question (see 'Uploading ground truth' below).

Adding a Reader Study

In order to set up a Reader Study, you need an account on grand-challenge.org. Click this link to sign up or log in with your existing account. Once you are logged in, navigate to the Reader Studies page on grand-challenge.org to add your reader study. If you do not see the button "Add a new reader study" please contact us to gain access. Some notes on the options:

  1. A square logo is required when you add your reader study (should have a resolution of at least 640x640 px).
  2. If you are unsure about the Workstation, select "Workstation CIRRUS Core (Public)". You will find more information on CIRRUS below.
  3. Workstation configurations control behaviour of the web workstation, for example which buttons are visible, which window level to show by default. This can be kept empty to get default options. You can browse and view the available configurations here
  4. Tick 'Is educational' if you want to teach your participants the right answer to each question. This requires uploading a ground truth later.
  5. Make it public or not. It is recommended that you do not make the study public at first. You can run a reader study, invite participants and collect the results without making the reader study public. Publication is only needed when you want to make your Reader Study visible to all grand-challenge users, who can then request access to it. If you already know your participants you can keep the Reader Study private and invite them yourself (see 'Inviting participants' below)

Uploading your data

When the basic configuration of your reader study is ready, it is time to collect your data and ensure it is in one of the following formats: .mha, .mhd, .raw, .zraw, .dcm, .tiff, .png, .jpeg and .jpg. You can upload your images by using the menu on the left hand side of the screen: Side menu → Cases as shown below. The status of your upload can be found on this page.

Alternatively, you could use the Grand Challenge API Client to upload your images through Python.

Reader Study configuration details (Hanging list)

A hanging list is used to structure all the different cases and images used in a reader study. In a hanging list one should specify which images are linked to which case. The default hanging list that is created each image is linked to one case. This could be perfectly fine for your case, but if you want to add a mask or a secondary image to each case the hanging list should be manually adjusted. The hanging list can be accessed by the Information → Update Settings. Here you see all the cases with the matching images. If you change "Tree" to "Code" in the dropdown menu in the header you can change the hanging list. Some examples are given below. Note that the image IDs are added to the hanging list automatically when you upload images. This is similar to the filenames.

Simple default hanging

By default, the simplest hanging list is chosen: one image per page. For two images this will look like this for example:

[
  {
    "main": "image1.mha"
  },
  {
    "main": "image2.mha"
  }
]

Side-by-side hanging

You can also have two images on one page. The following hanging shows two pages with two images side by side on each page:

[
  {
    "main": "sagittal_1.dcm",
    "secondary": "coronal_1.dcm"
  },
  {
    "main": "sagittal_2.dcm",
    "secondary": "coronal_2.dcm"
  }
]

Hanging with overlays

You can designate an image as an overlay on either the main or secondary viewport. The following shows a single page which has two images side by side, each with its own overlay:

[
  {
    "main": "sagittal_image_1.dcm",
    "main-overlay": "sagittal_mask_1.dcm",
    "secondary": "coronal_image_1.dcm",
    "secondary-overlay": "coronal_mask_1.dcm"
  }
]

Creating questions

Questions can be added for different purposes and with different types of input. Go to the Questions → Add a Question. You have to fill in multiple fields such as question text and help text, but most importantly the answer type. Here you have to choose what type of input you want for your question (bool, text, mask, bounding box, distance measurement etc.). The options there are mostly self-explanatory. Once a question has been answered by any participant it cannot be edited anymore. You can make the questions editable again by removing all answers (Users progress → Remove Answers). You can change the order of the questions by setting the 'Order' number in each question individually (Questions → Edit box next to each question). This will be the same order for all participants.

There is also the possibility to add a ground truth. This is only needed for Educational Reader Studies. Go to Ground Truth and follow the instructions there.

Going live

It is advised to first test your reader study. To be able to do this you should first add yourself as a reader (this is not automatically done). Go to Readers → add reader and add yourself with your own grand-challenge username. Once this is done click 'Launch this Reader Study' which opens a session in which you can test/use the reader study. Editors are still able to modify the reader study after it's being launched. You can clear your previous answers by going to Readers → Remove Answers.

When you are happy with your reader study, you can invite participants with the following steps:

  1. Each participant should register at grand-challenge.org and send you their username
  2. Go to the Readers → Add Readers and add their name
  3. Readers can then go to https://grand-challenge.org/reader-studies/your-reader-study/ to launch the Reader study

To see reader study completion per participant go to Readers.

If you uploaded a ground truth you can see details on the responses per case and per question from Statistics. Also at Leaderboard a leaderboard is shown with all participants ranked based on their performance.

If you want to export the results go to Export to download the data. All information is stored in different csv files which all can be downloaded separately. The first file that can be downloaded contains all the characteristics of the reader study such as settings, title and hanging list. The second file contains all questions and their settings. Next there are separate files containing all the images and answers. Note that these files only store up to 1000 images or answers. If the number of images or answers exceeds 1000 multiple csv files will be created. Alternatively, you could use the Grand Challenge API Client to download your results through Python.

Copy

There is also a "Copy" option which can be used to create a duplicate of the current reader study. You have the options to copy the entire reader study, or only certain parts of it (images, hanging list, case text, questions, readers or editors). This might be helpful when you want a different reader study where certain elements stay the same. For example you want to use the same images but different questions, or a complete different reader study for the same set of participants.

Delete

If you want to delete your reader study go to Information → Update Settings → scroll all the way down → Delete this reader study. Note that you are not able to undo this action, once the reader study is deleted it is deleted forever.