Create Your Own Reader Study
What is a Reader Study?
Reader Studies are a part of grand-challenge.org that can be used to collect annotations or score algorithm results for a set of medical images. They offer the following functionality:
- Ability to upload and host a set of medical images along with questions
- Invite participants to view the images and answer the questions
- Store and export results
There are two main Reader Study flavours:
-
In a Regular Reader Study each participant is asked to answer questions about a number of medical images and the answers are recorded.
-
An Educational Reader Study is meant as a teaching tool. It does everything a regular Reader Study does, and in addition gives the participant feedback on how they did on each question. An Educational Reader Study requires you to provide the correct answers for each question (see 'Uploading ground truth' below).
How to create a Reader Study
This section shows the typical running of a reader study from beginning to end.
Adding a Reader Study
In order to set up a Reader Study, you need an account on grand-challenge.org. Click this link to sign up or log in with your existing account. Once you are logged in, navigate to the Reader Studies page on grand-challenge.org to add your reader study. If you do not see the button "Add a new reader study" please contact us to gain access. Some notes on the options:
-
A square logo is required when you add your reader study.
-
If you are unsure about the Workstation, select "Workstation CIRRUS Core (Public)".
-
Tick 'Is educational' if you want to teach your participants the right answer to each question. This requires uploading a ground truth later.
-
Workstation configuration can be kept empty to get default options. You can also browse and view the available configs at grand-challenge.org/workstation-configurations/
Note on making public
It is recommended that you do not make the study public at first. You can run a reader study, invite participants and collect the results without making the reader study public. Publication is only needed when you want to make your Reader Study visible to all grand-challenge users, who can then request access to it. If you already know your participants you can keep the Reader Study private and invite them yourself (see 'Inviting participants' below)
Uploading your data
When the basic configuration of your reader study is ready, it is time to collect your data and ensure it is in one of the following formats: .mha, .mhd, .raw, .zraw, .dcm, .tiff, .png, .jpeg and .jpg
.
You can upload your images by using the menu on the left hand side of the screen: Side menu
->Cases
as shown below.
Alternatively, you could use the Grand Challenge API Client to upload your images through Python.
NOTE: Once the upload is complete, you can visualize the images only after adding yourself as a Reader: Side menu
-> Readers
-> add reader
and enter your grand-challenge username.
Creating questions
Go to the Side menu
-> Questions
-> Add a Question
. The options there are mostly self-explanatory.
NOTE: Once a question has been answered by any participant it cannot be edited anymore. You can make the questions editable again by removing all answers (Side menu
-> Readers
-> Remove Answers
)
Question order
You can order questions by setting the 'Order' number in each question individually (Side menu
-> Questions
-> Edit box next to each question). This will be the same order for all participants. Alternatively you can randomize the order for each participant by Side menu
-> top item
-> Update Settings
->Shuffle hanging list
.
Uploading ground truth
Ground truth is only needed for Educational Reader Studies. Side menu
-> Ground Truth
and follow the instructions there.
Testing your reader study
To see what your reader study looks like follow these steps:
- Add yourself as a reader go to the reader studies main page (top item in the left hand menu)
- click 'Launch this Reader Study'
You can clear your previous answers by going to Side menu
-> Readers
-> Remove Answers
Inviting participants
When you are happy with your reader study, you can invite participants with the following steps:
- Each participant should register at grand-challenge.org and send you their username
- Go to the
Side menu
->Readers
->Add Readers
and add their name - Readers can then go to
https://grand-challenge.org/reader-studies/<your-reader-study>/
to launch the Reader study
Monitoring progress
To see reader study completion per participant go to Side menu
-> Readers
If you uploaded a ground truth you can see details on the responses per question from Side menu
-> Statistics
Exporting reader study responses
Go to Side menu
-> Export
to download all responses as a csv file
Reader Study configuration details
Hanging
The hanging list determines which image is shown where on each page of your Reader Study. It can be accessed by the Side menu
-> top menu item (the name of your Reader Study)
-> Update Settings
.
The image id's are added to the hanging automatically when you upload images
Simple default hanging
By default, the simplest hanging is chosen: one image per page. For two images this will look like this for example:
[
{
"main": "6157ab48-1.3.6.1.4.1.14519.5.2"
},
{
"main": "e3d9e8b5-1.3.6.1.4.1.14519.5.1"
}
]
Side-by-side hanging
You can also have two images on one page. The following hanging shows two pages with two images side by side on each page:
[
{
"main": "6157ab48-1.3.6.1.4.1.14519.5.2",
"secondary": "e3d9e8b5-1.3.6.1.4.1.14519.5.1"
},
{
"main": "34787b23 - 31.3.6.1.4.1.14519.5.2",
"secondary": "sw49e8b5-1.3.6.1.4.1.14519.5.1"
}
]
Hanging with overlays
You can designate an image as an overlay on either the main or secondary viewport. The following shows a single page which has two images side by side, each with its own overlay:
[
{
"main": "6157ab48-1.3.6.1.4.1.14519.5.2",
"main-overlay": "34787b23 - 31.3.6.1.4.1.14519.5.2",
"secondary": "e3d9e8b5-1.3.6.1.4.1.14519.5.1",
"secondary-overlay": "sw49e8b5-1.3.6.1.4.1.14519.5.1"
}
]