Using the API for reader studies¶
In this tutorial we will focus on reader studies and show you how to upload cases to reader studies on our platform.
Remember that you need to request access prior to using a particular Reader study. You do not need to request permission if you are using your own reader study.
If you haven't installed gcapi yet, follow the instructions here.
Import necessary libraries:
import gcapi from pathlib import Path
Authenticate to Grand Challenge using your personal API token.
# authorize with your personal token token = 'my-personal-api-token' client = gcapi.Client(token=token)
Upload input to Reader studies on Grand Challenge¶
If you are working on a Reader study, you most likely need to upload the cases to the platform. This can be easily done via the API.
For this particular example, the data on your machine would be structured as follows:
mainfolder/ ├── patient1_folder │ ├──file_for_a_single_series ├── patient2_folder │ ├──file_for_a_single_series ├──patient3_folder │ ├──file_for_a_single_series ... ├──patientN_folder │ ├──file_for_a_single_series
Lets now list the files in the upload directory to check whether they can be identified:
# Specify the directory path which you want to upload from. upload_from_dir = Path(r"path\on\your\machine\with\data\for\reader\study") # Create list of files in the specified directory. files = sorted(f for f in upload_from_dir.rglob("*.*") if f.is_file()) print("Found", len(files), "cases for upload")
Create display sets¶
To create display sets for the readers to view, you need to provide a socket for each image. Within a display set, a socket needs to be unique. For example, one cannot have three generic-medical-images within a single display set.
In this example two sets are created and the display set contains three sockets: a ct-image, an airway-segmentation and some-score:
display_sets = [ { "ct-image": files[0], "airway-segmentation": files[1], "some-score": 2.1, }, { "ct-image": files[2], "airway-segmentation": files[3], "some-score": 7.9, }, ]
To upload cases and create the display sets, you will need to know the slug of the particular reader study you will work with. If the URL of your reader study on grand-challenge.org is https://grand-challenge.org/reader-studies/corads-score-exam/, the slug would be corads-score-exam.
# Specify the "slug" of the study you want to upload your data to. upload_reader_study_slug = "my-reader-study-slug" display_set_pks = client.add_cases_to_reader_study( reader_study=upload_reader_study_slug, display_sets=display_sets ) print("Uploaded cases and created display sets: " + display_set_pks)
Update the order of display sets¶
To update the order of a display set (in the example to 10), you can do the following:
client.reader_studies.display_sets.partial_update( pk="pk-of-display-set", order=10 )
Retrieve Reader Study results¶
You will need the slug of the reader study you wish to retrieve the results of. The response will contain general information, including the questions, of the reader study.
slug = "my-reader-study-slug" reader_study = client.reader_studies.detail(slug=slug)
Retrieve Reader Study Answers¶
You can retrieve only your answers or all answers for that reader study (if you are editor for that reader study) with the following code:
my_answers = list( client.reader_studies.answers.mine.iterate_all( params={"question__reader_study": reader_study["pk"]} ) ) answers = list( client.reader_studies.answers.iterate_all( params={"question__reader_study": reader_study["pk"]} ) )
If you would like to get the answers per display set, you can use the following code snippet.
# First get the display sets (add limit as parameter to iterate through the pages # if the number of display sets is more than 100). display_sets = client.reader_studies.display_sets.page( params={"reader_study": reader_study["pk"]} ) # Then get the answers for each display set. answers_per_display_set = {} for display_set in display_sets: answers_per_display_set[display_set.pk] = list( client.reader_studies.answers.iterate_all( params={"display_set": display_set.pk} ) )
Retrieve readable answers for (multiple) choice type questions¶
The answers for (multiple) choice type questions contain only the id of the chosen option, not the title. If you would like to add the title to the answers, you can do so by combining information from the reader study questions.
# Create a dictionary of the multiple choice questions with the question's api_url # as the key, and the options for the question as the value. # The options contain the readable title. choice_questions = { question.api_url: question for question in reader_study.questions if question.answer_type in ("Choice", "Multiple choice") } # Local function that will add the readable answer to the answer dictionary # for (multiple) choice questions. def add_answer_title(answer): if answer.question not in choice_questions: return answer options = choice_questions[answer.question].options if isinstance(answer.answer, list): # multiple choice answer.answer = list( o.title for o in options if o.id in answer.answer ) else: # choice answer.answer= list( o.title for o in options if o.id == answer.answer )[0] return answer # You can create a list for just the (multiple) choice type questions. choice_answers_readable = list( add_answer_title(a) for a in answers if a.question in choice_questions ) # Or create a list of readable items for all answers. answers_readable = list(str(a.answer) for a in answers)
Download answers saved as images¶
For questions of type Mask the answers are saved as images. You can download these as follows:
image_answers = list(a for a in answers if a.answer_image is not None) for i in image_answers: downloaded_files = client.images.download( url=i.answer_image, filename=Path("path/to/output") )