DRAGON Longformer Large Mixed-domain
About
Image Version:
d5eaf1fd-3890-49d6-93d8-5fea71e8e074
Last updated:
May 14, 2024, 9:30 a.m.
Interfaces
This algorithm implements all of the following input-output combinations:
|
Inputs |
Outputs |
1 |
-
- Description
- JSON object containing a configuration for NLP Tasks. The object must include the following properties: "jobid," "task_name," "input_name," "label_name," "recommended_truncation_side," and "version."
- Kind
- Anything
- Read from
-
/input/nlp-task-configuration.json
- Example
-
{
"jobid": 1010,
"version": "1.0",
"task_name": "Task101_Example_sl_bin_clf",
"input_name": "text",
"label_name": "single_label_binary_classification_target",
"recommended_truncation_side": "left"
}
NLP Task Configuration
-
- Description
- -
- Kind
- Anything
- Read from
-
/input/nlp-training-dataset.json
- Example
-
[
{
"uid": "Task101_case1",
"text": "Individual reports intermittent fever, symptoms of seasonal allergies. Patient presents with lesion of 3.8 mm. Patient presents with lesion of 14 cm. Case involves normal of 14 mm. Subject describes lesion of 11 cm. Individual reports normal of 3.7 mm. Individual reports normal of 16 mm. Individual reports other structure of size 18 mm. Subject describes lesion of 2.1 mm. Patient presents with normal of 19 mm. Patient presents with lesion of 3.7 mm. Patient presents with lesion of 15 cm. Individual reports lesion of 9.5 mm. Subject describes normal of 18 mm. Case involves lesion of 11 mm. Advise patient to increase fluid intake. ",
"single_label_binary_classification_target": true
},
{
"uid": "Task101_case3",
"text": "Case involves mild to moderate chest pain, possible bacterial infection. Subject describes lesion of 12 mm. Patient presents with lesion of 4.9 mm. Case involves normal of 3.8 mm. Patient presents with lesion of 14 mm. Individual reports normal of 19 mm. Patient presents with lesion of 14 mm. Patient presents with normal of 15 cm. Patient presents with lesion of 8.4 cm. Patient presents with lesion of 7.2 cm. Individual reports lesion of 2.8 mm. Subject describes normal of 1.4 mm. Patient presents with normal of 6.1 mm. Subject describes normal of 14 cm. Prescribe rest and over-the-counter medication. ",
"single_label_binary_classification_target": true
},
{
"uid": "Task101_case4",
"text": "Case involves sudden weight loss, potential thyroid disorder. Subject describes normal of 14 mm. Patient presents with lesion of 14 cm. Subject describes normal of 4.8 cm. Individual reports normal of 8.4 cm. Patient presents with lesion of 1.8 cm. Subject describes lesion of 1.8 cm. Patient presents with lesion of 2.4 mm. Individual reports other structure of size 17 cm. Individual reports other structure of size 11 mm. Case involves lesion of 5.0 mm. Case involves normal of 13 mm. Case involves lesion of 3.6 mm. Subject describes normal of 14 mm. Suggest dietary changes. ",
"single_label_binary_classification_target": true
}
]
NLP Training Dataset
-
- Description
- -
- Kind
- Anything
- Read from
-
/input/nlp-validation-dataset.json
- Example
-
[
{
"uid": "Task101_case5",
"text": "Subject describes sudden weight loss, potential thyroid disorder. Case involves normal of 13 mm. Subject describes normal of 20 cm. Subject describes normal of 9.4 mm. Individual reports lesion of 4.9 mm. Case involves normal of 12 mm. Patient presents with lesion of 2.8 cm. Subject describes lesion of 4.1 cm. Case involves lesion of 20 mm. Case involves normal of 16 mm. Subject describes lesion of 14 cm. Individual reports other structure of size 5.9 cm. Subject describes lesion of 14 mm. Individual reports other structure of size 8.6 cm. Subject describes normal of 4.7 mm. Advise patient to increase fluid intake. ",
"single_label_binary_classification_target": true
},
{
"uid": "Task101_case7",
"text": "Subject describes chronic fatigue, possible bacterial infection. Patient presents with normal of 20 mm. Case involves lesion of 9.0 mm. Subject describes normal of 16 mm. Subject describes normal of 14 mm. Subject describes normal of 19 mm. Case involves lesion of 17 mm. Patient presents with other structure of size 3.5 mm. Subject describes lesion of 4.7 mm. Prescribe rest and over-the-counter medication. ",
"single_label_binary_classification_target": false
},
{
"uid": "Task101_case13",
"text": "Case involves mild to moderate chest pain, early signs of diabetes. Case involves other structure of size 4.4 mm. Individual reports normal of 16 mm. Subject describes lesion of 14 cm. Individual reports other structure of size 8.1 mm. Case involves normal of 20 mm. Individual reports other structure of size 5.1 mm. Patient presents with normal of 13 mm. Case involves other structure of size 1.2 mm. Subject describes lesion of 12 mm. Patient presents with normal of 19 mm. Case involves lesion of 17 mm. Advise patient to increase fluid intake. ",
"single_label_binary_classification_target": true
}
]
NLP Validation Dataset
-
- Description
- -
- Kind
- Anything
- Read from
-
/input/nlp-test-dataset.json
- Example
-
[
{
"uid": "Task101_case0",
"text": "Patient presents with shortness of breath, likely viral infection. Subject describes other structure of size 13 mm. Case involves other structure of size 19 mm."
},
{
"uid": "Task101_case2",
"text": "Patient presents with shortness of breath, likely viral infection. Subject describes normal of 7.1 mm. Patient presents with normal of 6.7 mm. Case involves other structure of size 14 mm. Case involves lesion of 16 cm. Case involves normal of 4.4 cm. Patient presents with lesion of 9.1 mm. Subject describes normal of 12 mm. Prescribe rest and over-the-counter medication. "
},
{
"uid": "Task101_case9",
"text": "Case involves chronic fatigue, potential thyroid disorder. Case involves lesion of 5.2 mm. Patient presents with lesion of 8.5 mm. Patient presents with other structure of size 19 mm. Case involves normal of 3.6 mm. Individual reports normal of 14 mm. Individual reports normal of 13 mm. Individual reports other structure of size 20 mm. Case involves lesion of 10 mm. Patient presents with other structure of size 4.5 mm. Patient presents with normal of 1.9 mm. Patient presents with normal of 13 mm. Advise patient to increase fluid intake. "
}
]
NLP Test Dataset
|
-
- Description
- -
- Kind
- Anything
- Write to
-
/output/nlp-predictions-dataset.json
- Example
-
[
{
"uid": "Task101_case0",
"single_label_binary_classification": 0.8099229932
},
{
"uid": "Task101_case2",
"single_label_binary_classification": 0.9840752482
},
{
"uid": "Task101_case9",
"single_label_binary_classification": 0.0101556089
}
]
NLP Predictions Dataset
|
Challenge Performance
Model Facts
Summary
This algorithm is an adaptation of the DRAGON baseline (version 0.2.1), with pretrained foundational model
joeranbosma/dragon-longformer-large-mixed-domain. This algorithm is used for the pretraining experiment in the DRAGON manuscript [1], according to the pre-specified statistical analysis plan [2]. See dragon.grand-challenge.org/manuscript for the latest info on the DRAGON manuscript. Please cite the manuscript [1] when using this model.
[1] J. S. Bosma, K. Dercksen, L. Builtjes, R. André, C, Roest, S. J. Fransen, C. R. Noordman, M. Navarro-Padilla, J. Lefkes, N. Alves, M. J. J. de Grauw, L. van Eekelen, J. M. A. Spronck, M. Schuurmans, A. Saha, J. J. Twilt, W. Aswolinskiy, W. Hendrix, B. de Wilde, D. Geijs, J. Veltman, D. Yakar, M. de Rooij, F. Ciompi, A. Hering, J. Geerdink, H. Huisman, DRAGON Consortium. The DRAGON Benchmark for Clinical NLP. Under review.
[2] J. S. Bosma, K. Dercksen, L. Builtjes, R. André, C, Roest, S. J. Fransen, C. R. Noordman, M. Navarro-Padilla, J. Lefkes, N. Alves, M. J. J. de Grauw, L. van Eekelen, J. M. A. Spronck, M. Schuurmans, A. Saha, J. J. Twilt, W. Aswolinskiy, W. Hendrix, B. de Wilde, D. Geijs, J. Veltman, D. Yakar, M. de Rooij, F. Ciompi, A. Hering, J. Geerdink, H. Huisman, DRAGON Consortium (2024). DRAGON Statistical Analysis Plan (v1.0). Zenodo.
https://doi.org/10.5281/zenodo.10374512
Mechanism
For details on the pretrained foundational model, check out HuggingFace: joeranbosma/dragon-longformer-large-mixed-domain.
The following settings were used in the the DRAGON baseline:
model_name = "joeranbosma/dragon-longformer-large-mixed-domain"
per_device_train_batch_size = 1
gradient_accumulation_steps = 8
gradient_checkpointing = False
max_seq_length = 512
learning_rate = 1e-05
Validation and Performance
This model was tested on the 28 tasks in the DRAGON Benchmark for clinical NLP. For the full test leaderboard see here and for the performance of this model see here.
Uses and Directions
This algorithm was developed for research purposes only.
Warnings
You should anonymize your reports before uploading them to Grand Challenge.
Common Error Messages
Some logs are incorrectly seen as warnings, so each successful algorithm job will still say "Succeeded, with warnings". This warning can typically be ignored.
Information on this algorithm has been provided by the Algorithm Editors,
following the Model Facts labels guidelines from
Sendak, M.P., Gao, M., Brajer, N. et al.
Presenting machine learning model information to clinical end users with model facts labels.
npj Digit. Med. 3, 41 (2020). 10.1038/s41746-020-0253-3