Creating questions


Once you uploaded cases (i.e., images) to your reader study, it is time to define the questions that participants will need to answer for each of those cases.

Go to Questions → Add a Question.

Question text: The question that will be presented to the user, should be short. e.g. 'Is there pathology present in these images?'

Help text: This can be used to provide extra information or clarification to the reader about this question.

Answer type: The answer type defines the type of input for your question. Grand Challenge offers the following answer types:

Plain answer types:

  • bool
  • text
  • number
  • multiple choice

Annotation answer types:

  • mask
  • bounding box
  • distance measurement
  • polygon
  • point
  • line
  • angle
  • ellipse
  • three-point angle

The annotation-type answers are discussed in more detail here

Widget: Which widget should be presented to the reader in order to answer the question? Options depend on the chosen answer type above. More information is below.

Required: A question can be mandatory or optional, check the "Required" box accordingly.

Empty answer confirmation: Require an explicit confirmation when saving an empty answer to this question.

Image port: If the Answer type is an annotation you need to define on which image port it should be created. This will be the same image port for every case.

Default annotation color: For annotations you can specify which color should be used. If you leave it empty, either the color specified in your Viewer Configuration or the Cirrus default will be used.

Direction: The layout of the question, vertical means that the question text goes above the answer box, and horizontal means that the question text will be on the same row as the answer box.

Order: The order of the questions.

Interface: If a default answer will be provided in the Display Sets, select the relevant Interface here. How to set default answers is discussed in more detail here

Overlay segments: Relevant for segmentation-type answers. You can use this setting to specify the labels of your segmentation.

Look up table: Relevant for heatmap-type answers. You can use this setting to specify the LUT that will be applied to the heatmap.

💡Once a question has been answered by any participant it cannot be edited or deleted anymore. You can make the questions editable again by removing all answers (Users progress → Remove Answers) and any uploaded ground truth (Ground Truth → Delete ground truth).

Educational Reader Studies

You can create an Educational Reader Study by checking the is educational checkbox in the reader study settings. You also need to add the ground truth to monitor your user's performance. Go to Ground Truth and follow the instructions there. If ground truth has been added to a Reader Study, any answer given by a reader will be evaluated against the question-specific ground truth. The scores can then be compared on the leader board. Statistics are also available based on these scores: the average and total scores for each question as well as for each case are displayed in the statistics view.

💡It is not (yet) possible to include ground truth for annotation questions.

To streamline taking of the reader study you can enable the Instant verification checkbox in the reader study settings. Enabling this setting will allow the user to go through the reader study faster. The 'Save and continue' button will be replaced by a 'Verify and continue' button which will show the answer verification pop-up and allow the user to save and go to the next case upon dismissal.

Widgets

At this time, these are the following widgets.

Accept/Reject Findings

Only for: multiple annotations answer types (e.g. Multiple Points).

This widget allows the reader to explicitly accept or reject each finding presented to them in the reader study. This widget is very useful in combination with the need to review the outputs of an algorithm. Hence this widget requires default answers to be provided. An empty list can be provided when no findings are present: the reader is always able to add new findings. When the question is set to required, at least one finding should be provided by the reader or the answer to the question will be deemed invalid. They can either do this either by accepting at least one of the default findings or by creating a new finding.

Number input and Number Range

Only for: Number answer types

These widgets allow you to set minimum and maximum values to any numerical answer, and optional the step value for the Number Range widget:

  • 'Answer max'
  • 'Answer min'
  • 'Answer step size'

The effect of setting the maximum and minimum are straightforward constraints on the range. Setting the step to 1 (with an integer value for Answer min) will enforce integers whereas setting it to 0.5 will allow values such as 1, 1.5, or 2 but not 1.1. Setting a minimum will offset the step value. For example, a step of 0.3 with a minimum of 0.2 will allow values such as 0.2, 0.5, or 0.8.

Text Input and Text area

Only for: Text answer types

These widgets allow you to set minimum and maximum lengths for any textual answer, and offers the option to add a regular expression to validate the answer.

Radio Select and Select

Only for: Choice answer types

Present answers as a list of radio options or a dropdown with select option.

Checkbox Select Multiple and Select Multiple

Only for: Multiple choice answer types

Present answers as a list of checkboxes or a dropdown with multiple select options.

Previous section
Next section