---

About

Model Facts

Summary

This is a deep learning (DL) algorithm based on convolutional neural networks (CNNs) that accepts a low-dose chest CT and the coordinates of a pulmonary nodule as input to estimate the malignancy risk of the nodule. This algorithm was developed with pulmonary nodules from the National Lung Screening Trial (NLST) and validated externally with pulmonary nodules from the Danish Lung Cancer Screening Trial (DLCST).

Mechanism

Outcome	presence of malignancy in the pulmonary nodule(s)
Output	0 - 1 with 3 decimal values indicating the malignancy risk in the pulmonary nodule(s)
Target population	high-risk population eligible for lung cancer screening
Input data source	low-dose chest CT examination and coordinate(s) of pulmonary nodule(s)
Training data location and time-period	33 centers in US participating in NLST, CT examinations acquired between 2002 – 2004
Reference standard	histopathological confirmation or CT follow-up of at least 2 years
Model type	Convolutional neural network

** Calibration **¶

The Algorithm provides malignancy risk score(s) of the pulmonary nodule(s) between 0 - 1. Along with the raw_outputs from the algorithm in the JSON, calibrated_malignancy_risks are also provided for each nodule. The algorithm has been calibrated with Platt's scaling. A description of how the algorithm was calibrated is available here.

Validation and Performance

Dataset	Total nodules	Malignant nodules	AUC (95% CI)	Diameter in mm	Sensitivity (95% CI) at 90% specificity
NLST cohort (internal)	16077	1249 (7.8%)	.91 (.90, .92)	8.02 ± 5.28	891 of 1249; 71% (69, 74)
DLCST full cohort	883	65 (7.4%)	.93 (.89, .96)	7.09 ± 6.29	54 of 65; 84% (72, 94)
DLCST subset A	177	59 (33%)	.96 (.93, .99)	8.98 ± 8.75	54 of 59; 91% (82, 98)
DLCST subset B	175	59 (34%)	.86 (.80, .91)	12.82 ± 11.09	32 of 59; 54% (32, 78)

Uses and Directions

For research use only. The algorithm is intended to be used only on pulmonary nodules detected at low-dose chest CT examinations of high-risk participants eligible for lung cancer screening programs.

Benefits: Early detection of lung cancer may improve patient prognosis and potentially reduce treatment costs.

Target population and use case: High-risk population participating in lung cancer screening undergo annual low-dose CT scans. Radiologists reading chest CT scans will have to provide appropriate follow-up recommendations based on their assessment of pulmonary nodules.

General use: This model is intended to be used by radiologists for predicting the malignancy in pulmonary nodules detected at screening CT. The model is not a diagnostic for cancer and is not meant to guide or drive clinical care. This model is intended to complement other pieces of patient information in order to determine the appropriate follow-up recommendation.

Appropriate decision support: The model identifies pulmonary nodule X as at a high risk of being malignant. The referring radiologist reviews the prediction along with other clinical information and decides the appropriate follow-up recommendation for the patient.

Before using this model: Test the model retrospectively and prospectively on a diagnostic cohort that reflects the target population that the model will be used upon to confirm the validity of the model within a local setting.

Safety and efficacy evaluation: TBD

Warnings

Risks: Even if used appropriately, clinicians using this model can misdiagnose cancer. Delays in cancer diagnosis can lead to metastasis and mortality. Patients who are incorrectly treated for cancer can be exposed to risks associated with unnecessary interventions and treatment costs related to follow-ups.

Inappropriate Settings: This model was not trained on CT examinations of patients from a clinical or diagnostic setting. Do not use the model in the clinic without further evaluation. This model was trained to differentiate malignant pulmonary nodules from benign nodules on a high-risk population undergoing lung cancer screening with low-dose CT examinations. Do not use this model elsewhere.

Clinical rationale: The model is not interpretable and does not provide a rationale for high risk scores. Clinical end users are expected to place the model output in context with other clinical information to make the final determination of diagnosis.

Inappropriate decision support: This model may not be accurate outside of the target population and was developed on a high-risk population undergoing lung cancer screening. This model is not a diagnostic and is not designed to guide clinical diagnosis and treatment for malignant pulmonary nodules.

Generalizability: This model was primarily developed with pulmonary nodules from the National Lung Screening Trial and validated with pulmonary nodules from the Danish Lung Cancer Screening Trial. Do not use this model in an external setting without further evaluation.

Discontinue use if: Clinical staff raise concerns about the utility of the model for the intended use case or large, systematic changes occur at the data level that necessitates re-training of the model.

Common Error Messages

Information on this algorithm has been provided by the Algorithm Editors, following the Model Facts labels guidelines from Sendak, M.P., Gao, M., Brajer, N. et al. Presenting machine learning model information to clinical end users with model facts labels. npj Digit. Med. 3, 41 (2020). 10.1038/s41746-020-0253-3