Create Your Own Reader Study

What is a Reader Study?

Reader Studies are a part of grand-challenge.org that can be used to collect annotations or score algorithm results for a set of medical images. They offer the following functionality:

Ability to upload and host a set of medical images along with questions
Invite participants to view the images and answer the questions
Store and export results

There are two main Reader Study flavours:

In a Regular Reader Study each participant is asked to answer questions about a number of medical images and the answers are recorded.
An Educational Reader Study is meant as a teaching tool. It does everything a regular Reader Study does, and in addition gives the participant feedback on how they did on each question. An Educational Reader Study requires you to provide the correct answers for each question (see 'Uploading ground truth' below).

How to create a Reader Study

This section shows the typical running of a reader study from beginning to end.

Adding a Reader Study

In order to set up a Reader Study, you need an account on grand-challenge.org. Click this link to sign up or log in with your existing account. Once you are logged in, navigate to the Reader Studies page on grand-challenge.org to add your reader study. If you do not see the button "Add a new reader study" please contact us to gain access. Some notes on the options:

A square logo is required when you add your reader study.
If you are unsure about the Workstation, select "Workstation CIRRUS Core (Public)".
Tick 'Is educational' if you want to teach your participants the right answer to each question. This requires uploading a ground truth later.
Workstation configuration can be kept empty to get default options. You can also browse and view the available configs at grand-challenge.org/workstation-configurations/

Note on making public

It is recommended that you do not make the study public at first. You can run a reader study, invite participants and collect the results without making the reader study public. Publication is only needed when you want to make your Reader Study visible to all grand-challenge users, who can then request access to it. If you already know your participants you can keep the Reader Study private and invite them yourself (see 'Inviting participants' below)

Uploading your data

When the basic configuration of your reader study is ready, it is time to collect your data and ensure it is in one of the following formats: .mha, .mhd, .raw, .zraw, .dcm, .tiff, .png, .jpeg and .jpg. You can upload your images by using the menu on the left hand side of the screen: Side menu->Cases as shown below.

Alternatively, you could use the Grand Challenge API Client to upload your images through Python.

NOTE: Once the upload is complete, you can visualize the images only after adding yourself as a Reader: Side menu -> Readers -> add reader and enter your grand-challenge username.

Creating questions

Go to the Side menu -> Questions -> Add a Question. The options there are mostly self-explanatory.

NOTE: Once a question has been answered by any participant it cannot be edited anymore. You can make the questions editable again by removing all answers (Side menu -> Readers -> Remove Answers)

Question order

You can order questions by setting the 'Order' number in each question individually (Side menu -> Questions -> Edit box next to each question). This will be the same order for all participants. Alternatively you can randomize the order for each participant by Side menu -> top item -> Update Settings->Shuffle hanging list.

Uploading ground truth

Ground truth is only needed for Educational Reader Studies. Side menu -> Ground Truth and follow the instructions there.

Testing your reader study

To see what your reader study looks like follow these steps:

Add yourself as a reader go to the reader studies main page (top item in the left hand menu)
click 'Launch this Reader Study'

You can clear your previous answers by going to Side menu -> Readers -> Remove Answers

Inviting participants

When you are happy with your reader study, you can invite participants with the following steps:

Each participant should register at grand-challenge.org and send you their username
Go to the Side menu -> Readers -> Add Readers and add their name
Readers can then go to https://grand-challenge.org/reader-studies/<your-reader-study>/ to launch the Reader study

Monitoring progress

To see reader study completion per participant go to Side menu -> Readers

If you uploaded a ground truth you can see details on the responses per question from Side menu -> Statistics

Exporting reader study responses

Go to Side menu -> Export to download all responses as a csv file

Reader Study configuration details

Hanging

The hanging list determines which image is shown where on each page of your Reader Study. It can be accessed by the Side menu -> top menu item (the name of your Reader Study) -> Update Settings.

The image id's are added to the hanging automatically when you upload images

Simple default hanging

By default, the simplest hanging is chosen: one image per page. For two images this will look like this for example:

[
  {
    "main": "6157ab48-1.3.6.1.4.1.14519.5.2"
  },
  {
    "main": "e3d9e8b5-1.3.6.1.4.1.14519.5.1"
  }
]

Side-by-side hanging

You can also have two images on one page. The following hanging shows two pages with two images side by side on each page:

[
  {
    "main": "6157ab48-1.3.6.1.4.1.14519.5.2",
    "secondary": "e3d9e8b5-1.3.6.1.4.1.14519.5.1"
  },
  {
     "main": "34787b23 - 31.3.6.1.4.1.14519.5.2",
     "secondary": "sw49e8b5-1.3.6.1.4.1.14519.5.1"
  }
]

Hanging with overlays

You can designate an image as an overlay on either the main or secondary viewport. The following shows a single page which has two images side by side, each with its own overlay:

[
  {
    "main": "6157ab48-1.3.6.1.4.1.14519.5.2",
    "main-overlay": "34787b23 - 31.3.6.1.4.1.14519.5.2",
    "secondary": "e3d9e8b5-1.3.6.1.4.1.14519.5.1",
    "secondary-overlay": "sw49e8b5-1.3.6.1.4.1.14519.5.1"
  }
]