---

About

Model Facts

Summary

This algorithm is based on convolutional neural networks and uses conventional multi-view radiographs of the hand, wrist, and scaphoid to detect scaphoid fractures. It was designed for processing a radiographic study with an arbitrary number of series. It was developed in 2020-2021 at the Radboud University Medical Center and Jeroen Bosch Hospital in the Netherlands.

This algorithm description is adapted from the template proposed by Sendak, Gao, Brajer, and Balu (2020).

Mechanism

Algorithm description: The algorithm consists of four convolutional neural networks: a scaphoid localization and a laterality classification network, and two scaphoid fracture detection networks for processing frontal and lateral view radiographs separately. The first two networks are used to localize the scaphoid and to determine both its orientation (frontal view, including [ulnar-deviated] AP/PA and oblique, or lateral view) and laterality (left or right hand). The scaphoid is then extracted from the image and passed to the corresponding fracture detection network for fracture analysis. In this analysis, a fracture score is generated per anatomical region as defined by Ho and Wong (2011). These regions include the following: scaphoid tubercle (A1), distal articular (A2), distal 1/3 (B1), middle 1/3 (B2), proximal 1/3 (B3), and proximal pole (C). All processing steps are repeated for every input image and finally the maximum fracture score per region and per hand is selected.

Training data location and time-period: Picture archiving and communication systems of the Radboud University Medical Center and Jeroen Bosch Hospital in the Netherlands, radiographs acquired between 2003 - 2019.

Input:

• Data type: frontal view (i.e., anterior-posterior [AP], posterior-anterior [PA]), oblique view, and lateral view of the hand, wrist, or scaphoid.
• File format: DICOM or MHA file containing the Pixel Spacing Attribute (tag: 0028,0030) or Imager Pixel Spacing Attribute (tag: 0018,1164). The Photometric Interpretation Attribute (tag: 0028,0004) should be "MONOCHROME2" (minimum value is intended to be displayed as black).
• Target population: all patients with a sufficiently developed scaphoid bone (at least eight years old, preferably older than 18 years).

Output:

• Fracture score per anatomical region: scaphoid fracture score between 0 (fracture absent) and 1 (fracture present) for each of the six anatomical regions as defined by Ho and Wong (2011). For each fracture score it is indicated from which view the score was obtained. If multiple hands are depicted, then the scores are provided per hand.

At the moment of writing, Grand Challenge does not support the use of image series of arbitrary length as input and therefore the input has been limited to two images! Furthermore, the presentation of the algorithm output may be improved in the future.

Validation and Performance

Scaphoid localization network:

• Evaluation metrics: sensitivity, positive predictive value (PPV), and intersection over union (IoU).
• Ground truth: manually drawn bounding box of the scaphoid.
• Test set: 1117 wrist and scaphoid radiographs from the Jeroen Bosch Hospital.
• Results: the network achieved a sensitivity of 100% (1117/1117), PPV of 99.8% (1117/1119), and mean IoU of 0.958 ± 0.030.

Laterality classification network:

• Evaluation metrics: accuracy.
• Ground truth: manually classified laterality of the scaphoid.
• Test set: 1117 wrist and scaphoid radiographs from the Jeroen Bosch Hospital.
• Results: the network achieved an accuracy of 99.9% (1116/1117).

Scaphoid fracture detection networks :

• Evaluation metrics: sensitivity, specificity, positive predictive value (PPV), area under the receiver operating characteristic curve (AUC), and mean fracture localization precision (MLP). Automated image crops were used for evaluation.
• Ground truth: binary scaphoid fracture label (fracture present/absent) per anatomical region obtained through a consensus reading performed by two musculoskeletal (MSK) radiologists.
• Test set: 219 sets of hand, wrist, and scaphoid radiographs from the Jeroen Bosch Hospital (65 fracture cases, 154 non-fracture cases; each case represents one hand from one patient).
• Results: the network achieved a sensitivity of 72% (95% CI: 62, 83), specificity of 93% (95% CI: 88, 97), PPV of 81% (95% CI: 72, 91), AUC of 0.88 (95% CI: 0.82, 0.94), and MLP of 87% (95% CI: 78, 94).

Uses and Directions

Benefits: Automated scaphoid fracture diagnosis may diminish the risk of missing a fracture, reduce the costs of additional imaging studies and unnecessary therapy, speed up diagnosis, and allow earlier treatment.

Target population and use case: Patients clinically suspected of having a scaphoid fracture typically undergo conventional radiography. The algorithm preprocesses incoming wrist, hand, and scaphoid radiographs in the picture archiving and communication system, and may be able to assist residents, radiologists, or other physicians by acting either as a first or second reader, or as a triage tool that helps prioritizing worklists.

General use: This algorithm is intended to be used by radiologists for identifying scaphoid fractures on conventional radiographs of the hand, wrist, or scaphoid. It is not a diagnostic for scaphoid fractures and is not meant to guide or drive clinical care. It should only be used to complement other pieces of patient information related to scaphoid fractures as well as a physical evaluation to determine the need for scaphoid fracture treatment.

Appropriate decision support: The algorithm predicts whether any scaphoid fractures are present. The radiologist examines the predicted scaphoid fracture scores along with other clinical information to determine if a scaphoid fracture is present.

Before using this algorithm : Test the algorithm retrospectively on a diagnostic cohort that reflects the target population that the algorithm will be used upon to confirm validity of the algorithm within a local setting.

Safety and efficacy evaluation: An observer study with retrospectively collected data was conducted and showed that the algorithm was able to detect scaphoid fractures just as well as five MSK, achieving a comparable AUC. When the algorithm assisted the MSK radiologists, it improved five out of ten pairs of inter-observer Cohen’s κ agreement (average increase of 36.2%) and reduced the reading time of four MSK radiologists (average reduction of 49.4%), but no improvements were found in sensitivity, specificity, PPV, and AUC for the majority of MSK radiologists. Future research should evaluate the impact of AI assistance on diagnostic performance, clinical decision making, and patient outcomes in a randomized clinical trial involving both radiologists and non-radiologists.

Warnings

Risks: Even if used appropriately, radiologists using this algorithm can misdiagnose scaphoid fractures. Delays in diagnosing a scaphoid fracture put patients at a greater risk of developing a non-union fracture, which may lead to complications such as avascular necrosis, carpal instability, osteoarthritis, and ultimately functional loss. False positive diagnoses lead to unnecessary wrist immobilization, which increases health expenditure and decreases patients’ productivity.

Inappropriate settings: This algorithm was not trained or evaluated on radiographs in which the scaphoid is incompletely depicted, obstructed by casts or implants, excessively damaged or malformed, (partially) resected, underdeveloped (children). Do not use the algorithm when any of these circumstances apply. Only use radiographs that have been made specifically for fracture diagnosis and depict a hand or wrist in a neutral position (i.e. fingers pointing upwards).

Clinical rationale: The algorithm provides a fracture score per scaphoid region. These scores are only interpretable as long as the input radiograph is similar to the radiographs used for training the algorithm. Clinical end users are expected to place the algorithm output in context with other clinical information to make final determination of diagnosis.

Inappropriate decision support: This algorithm may not be accurate outside of the target population (primarily adult patients). This algorithm is not a diagnostic and is not designed to guide clinical diagnosis and treatment for scaphoid fractures.

Generalizability: This algorithm was evaluated within the local setting of the Radboud University Medical Center and Jeroen Bosch Hospital. Do not use this algorithm in an external setting without further evaluation.

Discontinue use if: Clinical staff raise concerns about the utility of the algorithm for the indicated use case or large, systematic changes occur at the data level that necessitates re-training of the algorithm.

Common Error Messages

"Uneven pixel spacing encountered. Image scaling might not be accurate": The uploaded image has a different pixel spacing in the x and y direction. The algorithm uses the pixel spacing information to rescale the image in order to normalize the scaphoid size before segmentation. Provided that the pixel spacing information is inaccurate, the scaling operation may result into an unnatural looking image.

"Pixel spacing of (1, 1) encountered. Pixel spacing information could be wrong or missing": The uploaded image most likely does not contain valid pixel spacing information. Accurate pixel spacing information is required to normalize the scaphoid size before segmentation.

"Bounding box is relatively small or large. Possible scaphoid detection failure": The height or width of the bounding box derived from the generated scaphoid segmentation mask is outside three standard deviations from the mean height or width of the bounding boxes in the training data. This may indicate a failure of the segmentation network to localize the scaphoid.

"Cannot classify the image due to scaphoid detection failure": The scaphoid localization network failed to detect and localize any part of the scaphoid and hence no bounding box could be generated. Therefore, there is no region of interest to process by the fracture detection network.

Information on this algorithm has been provided by the Algorithm Editors, following the Model Facts labels guidelines from Sendak, M.P., Gao, M., Brajer, N. et al. Presenting machine learning model information to clinical end users with model facts labels. npj Digit. Med. 3, 41 (2020). 10.1038/s41746-020-0253-3