When simpler models win: A large-scale computational benchmark of lexical and transformer NLP pipelines for predicting medication effectiveness

Al-Turkey, Doaa and Nassr, Rasheed Mohammad and Grieshofer, Tatiana (2026) When simpler models win: A large-scale computational benchmark of lexical and transformer NLP pipelines for predicting medication effectiveness. Journal of Computational Science, 101. p. 103017. ISSN 1877-7503

Preview

Text 1-s2.0-S1877750326002358-main.pdf - Published Version Available under License Creative Commons Attribution. Download (6MB)

Abstract

Patient-generated drug reviews provide a scalable source of real-world evidence on perceived medication effectiveness, yet robust and reproducible modelling benchmarks remain limited. This study presents a quantitative evaluation of seven natural language processing pipelines for predicting drug-level effectiveness ratings from patient reviews on a 1–10 scale. Reviews were aggregated at the medication level by concatenating texts per drug and constructing a stabilised drug-level target using Bayesian shrinkage (α = 10.0). Models spanning sparse lexical baselines, dense embedding pipelines, frozen biomedical transformers, and a compute-aware fine-tuned transformer baseline (that is, one tuned under an explicitly bounded compute budget) were assessed using Repeated K-Fold cross-validation on Dataset A (5 folds × 2 repeats; out-of-fold prediction) and external validation on four independent datasets (A→B-E). Performance was measured using RMSE and MAE, with R², Pearson correlation, Spearman correlation, and calibration slopes, intercepts, and plots reported to characterise association and reliability under dataset shift. Across internal validation and most external evaluations, TF-IDF with Ridge regression achievedTF-IDF with linear support vector regression (SVR) achieved the strongest overall accuracy (Dataset A internal RMSE = 0.336; R² = 0.813) and generalised well, particularly on Dataset and generalised consistently, achieving the lowest error on every external corpus, including Dataset B (RMSE = 0.438), the only external dataset sharing no reviews with the training data. External results are reported after removing reviews duplicated from the training corpus, which affected 97.8 per cent of Dataset C, 98.9 per cent of Dataset D and 28.7 per cent of Dataset E. Transformer-based approaches, including SBERT and frozen biomedical encoders, did not outperform lexical baselines, and a transparently tuned ClinicalBERT regression baseline using a learning-rate sweep {1e-5, 2e-5, 3e-5} with early stopping did not yield consistent gains under external shift. The best lexical baseline achieved consistently lower RMSE than the best transformer pipeline in every one of the five evaluation settings, with a median RMSE gap of 0.19 rating points. A paired non-parametric test is reported alongside this descriptive pattern (Wilcoxon signed-rank, p = 0.031), but with only five paired settings this is the minimum attainable p-value and it is therefore treated as a caveat-bearing supplement rather than as confirmatory evidence. These findings indicate that, under drug-level aggregation and compute-aware tuning, frequency-based lexical signals capture much of the predictive structure in patient reviews. The benchmark provides a reproducible framework for evaluating drug-review-based effectiveness prediction with internal and external validation and reliability diagnostics relevant to pharmacovigilance and digital health analytics.

Item Type:	Article
Identification Number:	10.1016/j.jocs.2026.103017
Dates:	Date Event 24 August 2026 Accepted 27 August 2026 Published Online
Uncontrolled Keywords:	Patient-generated drug reviews, Drug-level effectiveness prediction, Natural language processing, TF-IDF, Support vector regression, ClinicalBERT, External validation
Subjects:	CAH11 - computing > CAH11-01 - computing > CAH11-01-01 - computer science CAH19 - language and area studies > CAH19-01 - English studies > CAH19-01-01 - English studies (non-specific)
Divisions:	Architecture, Built Environment, Computing and Engineering > Computer Science Arts > English and Media > English
Depositing User:	Gemma Tonks
Date Deposited:	13 Nov 2026 16:20
Last Modified:	13 Nov 2026 16:20
URI:	https://www.open-access.bcu.ac.uk/id/eprint/17314

Actions (login required)

View Item