Will label "other" be evaluated ? (Category 2)

According to "Submission Instructions", label "other" is mapped into "7" as ont of the surgical step, but it is hardly seen in the given .csv file.

I would like to know whether the label "other" is included in the f1-score evaluation. Thanks.

Yes, the "other" label will be included in the f1-score evaluation. Any unannotated parts of the videos are to be treated as "other" label so that will result in "other" class being one of the bigger ones. We have added clarifying notes in the data description and submission instructions page as well.

Best, SurgVU Organizing Committee

Thank you for your reply!

There is one more question that I would like to ask, how is the weight in mean weighted f1-score computed? Is it based on the frequency of each labels in the training data or the testing data (that are hidden from the participants)?

The weighted f1-score will only be based on the testing set.

Best, SurgVU team