---

About

Model Facts

Summary

This deep learning based algorithm detects pulmonary nodules in a thorax or thorax-abdomen CT scan in a routine clinical setting. The algorithm was developed with 888 CT scans from the LIDC-IDRI archive and 1,102 CT scans from Radboud University Medical Center. The algorithm was evaluated on an internal test set of 100 CT scans from Radboud University Medical Center and external test set of 100 CT scans from Jeroen Bosch Hospital.

Mechanism

Input:

• Data type: thorax or thorax-abdomen CT(A) scan with a maximum slice thickness of 3 mm.
• File format: MHA or DICOM
• Target population: Adult patients (18 years or older)

Output:

• Nodule locations: world coordinates (X,Y,Z)
• Nodule likelihood: nodule likelihood probability between 0 and 1.
• Overlay: points that indicate the nodule locations with a clinically acceptable false positive rate (i.e., an average of 1 false positive detection per scan).

Model type: Convolutional neural network Training data location and time period: CT examinations from seven academic centers in the United States and eight medical imaging companies (LIDC-IDRI dataset, exact locations and acquisition dates unknown, acquired before 2011). Additional training data obtained from Radboud University Medical Center, examinations acquired in 2017.

Reference standard: A panel of five thoracic radiologists independently labelled all pulmonary nodules. Two additional radiologists verified the nodule malignancy status and searched for any missed cancers using data from the Netherlands Cancer Registry. All malignant nodules were included and confirmed by either confirmed by histological examination, cytology testing, or follow-up imaging. Benign nodules were included if they were detected by at least three thoracic radiologists (majority-vote). A nodule was considered benign if it was stable and not followed by a cancer diagnosis within two years.

Validation and Performance

The table below shows the model sensitivity at 7 predefined false positive rates (0.125, 0.25, 0.5, 1, 2, 4, and 8 average false positives per scan) on a Free Receiver Operating Characteristic (FROC) curve on the complete dataset (Radboudumc and JBZ combined). The competition performance metric (CPM) is the average sensitivity at all false positive rates.

Internal (hospital A)

	Count	0.125	0.25	0.5	1	2	4	8	CPM
Nodules ≥ 3 mm	319	60.8	72.1	82.1	90.9	95.6	97.5	99.1	85.4
Nodules ≥ 4 mm	250	72.4	80.8	89.2	92.8	96.8	98	99.2	89.9
Nodules ≥ 5 mm	188	77.1	83	89.9	91.5	96.3	97.9	99.5	90.7
Actionable nodules	63	77.8	82.5	88.9	92.1	96.8	98.4	100	90.9
Primary cancers	27	77.8	81.5	92.6	92.6	96.3	100	100	91.5
Metastases	165	60	68.5	78.2	90.3	95.2	96.4	98.8	83.9

External (hospital B)

	Count	0.125	0.25	0.5	1	2	4	8	CPM
Nodules ≥ 3 mm	303	68.6	77.3	86.5	92.4	94.4	96.7	97.4	87.6
Nodules ≥ 4 mm	262	76.3	83.6	90.1	94.3	96.2	97.3	97.7	90.8
Nodules ≥ 5 mm	215	79.1	85.1	90.7	94.4	96.7	97.7	98.1	91.7
Actionable nodules	87	72.4	82.8	90.8	94.3	96.6	97.7	98.9	90.5
Primary cancers	32	87.5	90.6	93.8	96.9	96.9	96.9	96.9	94.2
Metastases	113	76.1	79.8	88.5	92	93.8	96.5	96.5	89

Uses and Directions

Benefits: This model may aid the radiologists obtaining a timely diagnosis of lung cancer and monitor metastasized cancer for adequate treatment.

Target population and use case: Any adult patient (18 years or older) who would be eligible for a thorax or thorax-abdomen CT scan at the hospital. The model may assist residents, radiologists, or other physicians by acting as a concurrent reader for the detection of lung nodules in CT scans.

General use: This model is intended for detecting intrapulmonary nodules in thorax or thorax-abdomen CT scans for research purposes only. The model has not been certified for clinical use. Radiologists can decide whether follow-up is needed based on these detections and their own findings in correspondence to established nodule management guidelines (such as BTS or Fleischner).

Appropriate decision support: This model returns the locations of potential lung nodules. The referring radiologist checks for additional nodules in the scan and assesses all them along with other clinical information. The radiologist decides on appropriate follow-up recommendation for the patient.

Before using this model: Test the model retrospectively and prospectively on a diagnostic cohort that reflects the target population that the model will be used upon to confirm the validity of the model within a local setting.

Safety and efficacy evaluation: TBD

Warnings

Risks: Even if used appropriately, clinicians using this model may miss relevant lesions that turn out to be lung cancer or pulmonary metastases. Lung cancer can also manifest as thick-walled cysts, airway lesions, or large masses (> 3 cm), which were not labelled in the training data. Delays in lung cancer diagnosis can lead to metastasis and mortality. Patients who are incorrectly treated for cancer can be exposed to risks associated with unnecessary interventions and treatment costs related to follow-ups.

Inappropriate settings: This model was trained on CT scans of the thorax or thorax-abdomen region. Do not use the algorithm for CT scans that only partly depict the lungs, such as CT scans of the abdomen, neck, thoracic spine, or heart. Do not use the algorithm for processing CT scans from patients with extensive interstitial diseases (that involve nodular fibrosis), severe consolidations (e.g., pneumonia), or substantial atelectasis.

Clinical rationale: This model does not provide a rationale for the detected lung nodule candidates. Clinical end users are expected to place the model output in context with other clinical information to make the final determination of diagnosis.

Inappropriate decision support: This model may not be accurate outside of the target population and was developed on a cohort of a routine clinical population. This model is not a diagnostic and is not designed to guide clinical diagnosis and treatment for malignant pulmonary nodules.

Generalizability: This model was evaluated within the local setting of the Radboud University Medical Center and Jeroen Bosch Hospital. Do not use this model in an external setting without further evaluation.

Discontinue use if: Clinical staff raise concerns about the utility of the model for the intended use case or large, systematic changes occur at the data level that necessitates re-training of the model.

Common Error Messages

Information on this algorithm has been provided by the Algorithm Editors, following the Model Facts labels guidelines from Sendak, M.P., Gao, M., Brajer, N. et al. Presenting machine learning model information to clinical end users with model facts labels. npj Digit. Med. 3, 41 (2020). 10.1038/s41746-020-0253-3