Deep Learning to Refine the Identification of High-Quality Clinical Research Articles from the Biomedical Literature: Performance Evaluation
Lokker, Cynthia and Bagheri, Elham and Abdelkader, Wael and Parrish, Rick and Afzal, Muhammad and Navarro, Tamara and Cotoi, Chris and Germini, Federico and Linkins, Lori and Haynes, R. Brian and Chu, Lingyang and Iorio, Alfonso (2023) Deep Learning to Refine the Identification of High-Quality Clinical Research Articles from the Biomedical Literature: Performance Evaluation. Journal of Biomedical Informatics. p. 104384. ISSN 1532-0464
Preview |
Text
BioBert-Afzal.pdf - Accepted Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (1MB) |
Abstract
Background
Identifying practice-ready evidence-based journal articles in medicine is a challenge due to the sheer volume of biomedical research publications. Newer approaches to support evidence discovery apply deep learning techniques to improve the efficiency and accuracy of classifying sound evidence.
Objective
To determine how well deep learning models using variants of Bidirectional Encoder Representations from Transformers (BERT) identify high-quality evidence with high clinical relevance from the biomedical literature for consideration in clinical practice.
Methods
We fine-tuned variations of BERT models (BERTBASE, BioBERT, BlueBERT, and PubMedBERT) and compared their performance in classifying articles based on methodological quality criteria. The dataset used for fine-tuning models included titles and abstracts of >160,000 PubMed records from 2012-2020 that were of interest to human health which had been manually labeled based on meeting established critical appraisal criteria for methodological rigor. The data was randomly divided into 80:10:10 sets for training, validating, and testing. In addition to using the full unbalanced set, the training data was randomly undersampled into four balanced datasets to assess performance and select the best performing model. For each of the four sets, one model that maintained sensitivity (recall) at ≥99% was selected and were ensembled. The best performing model was evaluated in a prospective, blinded test and applied to an established reference standard, the Clinical Hedges dataset.
Results
In training, three of the four selected best performing models were trained using BioBERTBASE. The ensembled model did not boost performance compared with the best individual model. Hence a solo BioBERT-based model (named DL-PLUS) was selected for further testing as it was computationally more efficient. The model had high recall (>99%) and 60% to 77% specificity in a prospective evaluation conducted with blinded research associates and saved >60% of the work required to identify high quality articles.
Conclusions
Deep learning using pretrained language models and a large dataset of classified articles produced models with improved specificity while maintaining >99% recall. The resulting DL-PLUS model identifies high-quality, clinically relevant articles from PubMed at the time of publication. The model improves the efficiency of a literature surveillance program, which allows for faster dissemination of appraised research.
Item Type: | Article |
---|---|
Identification Number: | 10.1016/j.jbi.2023.104384 |
Dates: | Date Event 3 May 2023 Accepted 8 May 2023 Published Online |
Uncontrolled Keywords: | bioinformatics, machine learning, evidence-based medicine, literature retrieval, medical informatics, Natural Language Processing |
Subjects: | CAH00 - multidisciplinary > CAH00-00 - multidisciplinary > CAH00-00-00 - multidisciplinary CAH11 - computing > CAH11-01 - computing > CAH11-01-01 - computer science CAH11 - computing > CAH11-01 - computing > CAH11-01-03 - information systems |
Divisions: | Faculty of Computing, Engineering and the Built Environment > College of Computing |
Depositing User: | Muhammad Afzal |
Date Deposited: | 23 May 2023 15:32 |
Last Modified: | 08 May 2024 03:00 |
URI: | https://www.open-access.bcu.ac.uk/id/eprint/14398 |
Actions (login required)
View Item |