Creating questions

Once you uploaded cases (i.e., images) to your reader study, it is time to define the questions that participants will need to answer for each of those cases.

Go to Questions β†’ Add a Question. Fill in a question text and an optional help text and decide on an answer type. The answer type defines the type of input for your question. Grand Challenge offers the following answer types:

Plain answer types:

  • bool
  • text
  • number
  • multiple choice

Annotation answer types:

  • mask
  • bounding box
  • distance measurement
  • polygon
  • point

The annotation type answers are discussed in more detail here

A question can be mandatory or optional, check the "Required" box accordingly.

Note that once a question has been answered by any participant it cannot be edited or deleted anymore. You can make the questions editable again by removing all answers (Users progress β†’ Remove Answers).

You can change the order of the questions by setting the 'Order' number in each question individually (Questions β†’ Edit box next to each question). This will be the same order for all participants.

If you are setting up an Educational Reader Study, you also need to add the ground truth to monitor your users performance. Go to Ground Truth and follow the instructions there. If ground truth has been added to a Reader Study, any answer given by a reader will be evaluated against the question-specific ground truth. The scores can then be compared on the leader board. Statistics are also available based on these scores: the average and total scores for each question as well as for each case are displayed in the statistics view.

πŸ’‘It is not (yet) possible to include ground truth for annotation questions.