Skip to main content

Open Access Comprehensive analysis of responses from ChatGPT to consumer inquiries regarding over-the-counter medications

Background: The use of generative artificial intelligence (AI) applications such as ChatGPT is becoming increasingly popular. In Japan, consumers can purchase most over-the-counter (OTC) drugs without having to consult a pharmacist, so they may ask generative AI applications which OTC drugs they should purchase. This study aimed to systematically evaluate responses from ChatGPT to consumer inquiries about various OTC drugs. Methods: We selected 22 popular OTC drugs and 12 typical consumer characteristics, including physical and disease conditions and concomitant medications. We input a total of 264 questions (i. e., all combinations of drugs and characteristics) to ChatGPT in Japanese, asking whether it is safe for consumers with each characteristic to take these OTC drugs. We used the generic name for 10 of the 22 drugs and the brand name for the remaining 12. Responses were evaluated based on the following three criteria: 1) coherence between the question and response, 2) scientific correctness, and 3) appropriateness of the instructed actions. When we received a response that satisfied all three criteria, we input the exact same question on a different day to assess reproducibility. Results: The proportions of ChatGPT's answers that satisfied criteria 1, 2, and 3 were 79.5%, 54.5%, and 49.6%, respectively. However, the proportion of responses that satisfied all three criteria was only 20.8% (55/264); 61.8% (34/55) of these responses were reproduced when the same question was input again on a different day. Compared with questions using generic names, those using brand names resulted in lower coherence and scientific correctness. Among the 12 characteristics, the appropriateness of the instructed actions tended to be lower in responses to questions about driving and concomitant medications. Conclusions: Our study revealed that ChatGPT was less accurate in its responses and less consistent in its instructed actions compared with the package inserts. Our findings suggest that Japanese consumers should not consult ChatGPT regarding OTC medications, especially when using brand names.

Document Type: Research Article

Affiliations: 1: Division of Clinical Pharmacy , Keio University Faculty of Pharmacy; Corresponding author: Keisuke Kiyomiya, Division of Clinical Pharmacy, Keio University Faculty of Pharmacy, 1-5-30 Shibakoen, Minato-ku, Tokyo 105-8512, Japan, Email: [email protected] 2: Division of Clinical Pharmacy, Keio University Faculty of Pharmacy; Department of Pharmacy, Keio University Hospital; 3: Division of Clinical Pharmacy, Keio University Faculty of Pharmacy; Department of Pharmacy, Keio University Hospital; Department of Clinical Pharmacy, Keio University School of Medicine, Tokyo, Japan

Publication date: February 1, 2024

More about this publication?
  • Pharmazie is a leading journal in the field of pharmaceutical sciences. As a peer-reviewed scientific journal, Pharmazie is regularly indexed in the relevant databases like Web of science, Journal Citation Reports and many others. The journal is open for submissions from the whole spectrum of pharnaceutical sciences including Pharmaceutical Chemistry, Experimental and Clinical Pharmacology, Drug Analysis, Pharmaceutics, Pharmaceutical Biology, Clinical Pharmacy etc.
  • Information for Authors
  • Submit a Paper
  • Subscribe to this Title
  • Ingenta Connect is not responsible for the content or availability of external websites
  • Access Key
  • Free content
  • Partial Free content
  • New content
  • Open access content
  • Partial Open access content
  • Subscribed content
  • Partial Subscribed content
  • Free trial content