A large language model based approach to providing personalised health-related information for osteoporosis patients

Wilson, Andrew and Adesina, Anuoluwapo and Jiao, Yunzheng and Douglas, Karen (2024) A large language model based approach to providing personalised health-related information for osteoporosis patients. Annals of Rheumatic Disease, 83 (S1). p. 204. ISSN 0003-4967

Full text not available from this repository.

Official URL: https://ard.bmj.com/content/83/Suppl_1/204.1

Abstract

Background: The Internet has long been recognised as a medium that patients use to gather information about their medical conditions [1]. The wealth of information it contains and uncertainty about its credibility can lead to the lack of personalisation for the patient, resulting in time consuming searches for the information that the patient wants.

Objectives: In order to improve patient information management this project created a mobile device-based large language model (LLM) chat-bot that could provide non-pharmacological and generalised recommendations for people living with osteoporosis.

Methods: The system implements Retrieval Augmented Generation (RAG) pattern and consists of the following foundational components: a chat-bot interface designed to interact with users; a gpt-3.5-turbo LLM chain with generative question-answering (G-Q-A) capabilities; a database to store vector embeddings; a retriever using maximum marginal relevance (MMR) algorithm to query the vector embeddings. All information was derived from clinically approved sources including Royal Osteoporosis Society, National Osteoporosis Society, National Osteoporosis Guideline Group, UK, International Osteoporosis Foundation (IOF) and National Osteoporosis Foundation. The system runs on Android mobile devices. The system was evaluated based upon technical metrics (correct responses to 97 pre-generated question-and-answer pairs), its ability to handle unethical questions or questions outside the defined domain as well as for overall usability (System Usability Scale questionnaire [SUS]) which rates usability on a scale of 0 (poor) to 100 (excellent). The latter was done with the help of general users who were healthy volunteers age range:18-24 (n:1); 25-34 (n:17); 35-44(n:1) and 55+ (n:1) as as well as medical professionals (n:4).

Results: Evaluation revealed that gpt-3.5-turbo scored 96% for predicting correct answers on the evaluation dataset of questions. Unethical and non-domain specific questions were correctly filtered out. Usability was scored as excellent by both medical professionals (83±11.6) and general users (85±12.9).

Conclusion: With growing and fast pace advances in artificial intelligence the authors have demonstrated that LLM-Chatbots can be used to manage large amounts of health-related information in a reliable way. It has the potential to provide personalise information to patient queries using an easy to use computer-based question and answering system. Further evaluations will be needed with the help of osteoporosis patients in order to further refine it.

REFERENCES: [1] Wilson AS, Kitas GD, Lewellyn P, Carruthers DM, Cheseldine DC, Harris S, Bacon PA, Huisson AP and Young SP (2001). Provision of Internet Based Rheumatology Education. Rheumatology 40(6): 645-51.

Item Type:	Article
Dates:	Date Event 27 March 2024 Accepted 11 June 2024 Published Online
Subjects:	CAH11 - computing > CAH11-01 - computing > CAH11-01-01 - computer science CAH11 - computing > CAH11-01 - computing > CAH11-01-05 - artificial intelligence
Divisions:	Faculty of Computing, Engineering and the Built Environment > College of Computing
Depositing User:	Andrew Wilson
Date Deposited:	21 Jun 2024 14:11
Last Modified:	21 Jun 2024 14:11
URI:	https://www.open-access.bcu.ac.uk/id/eprint/15571