SA1 - The Effectiveness of Artificial Intelligence-Based Virtual Assistants in Addressing Frequently Encountered Inquiries from Patients in Oral Surgery Clinics.

Abstract: Statement of Problem: Artificial intelligence (AI) is one of most rapidly evolving technologies, and humans are constantly discovering novel ways of utilizing it to enhance accuracy, efficiency and decision making in many industries. It has a number of applications in healthcare, including increasing the sensitivity and specificity of imaging modalities, creating personalized treatment plans for patients, and more accurate patient monitoring. With inevitable integration into cell phones on the horizon, comprehensive evaluation comparing the effectiveness of different chatbots in this context is warranted. This study aims to assess the effectiveness and practicality of four text-based artificial intelligence virtual assistants, namely ChatGPT-3.5, ChatGPT-4, Google Gemini (formerly Google Bard), and Microsoft Copilot (formerly Bing AI), in addressing frequently encountered inquiries from patients in oral surgery clinics. By analyzing the utility efficacy of these artificial intelligence systems, we intend to provide useful insights into their potential applications in the field of oral and maxillofacial surgery from a patient communication perspective.Materials/

Methods: A cohort of fourteen board certified oral and maxillofacial surgeons participated in the assessment(15 min.) of a predetermined set of fifteen questions that were posed to the chatbots(Figure 1). To gauge the performance of each artificial intelligence system in addressing the queries, a modified Likert scale was employed as a standardized evaluation tool(Figure 2). This method allowed for an objective assessment of the chatbots' proficiency in responding to the questions posed. Additionally, the evaluators were asked to answer one additional question, which was to give a score of 1-6 evaluating the chatbots’ overall performance(Figure 2).Methods of Data Analysis: The surgeon’s determinations were subsequently documented on an Excel spreadsheet for statistical evaluation. One-way analysis of variance (ANOVA) and subsequent Bonferroni post-hoc analyses were conducted to ascertain the mean responses of the devices across the investigators(Figure 1). The primary analysis aimed to compare the aggregated mean scores across the four devices, where lower scores were indicative of increased efficiency and utility. The secondary analysis aimed to compare the combined mean values of all the inquiries among the four devices, assessing the potential variability in device performance and responsiveness among individual investigators(Figure 3).Results/Outcomes: ChatGPT-4 emerged as the most effective tool, with the lowest mean score of 2.05, indicating high accuracy and reliability in responding to OMFS-related queries. ChatGPT-3.5 followed with a score of 2.14, then Google Gemini 2.29, and Microsoft Copilot 2.58. Notably, ChatGPT-4 demonstrated the least variability in responses and was deemed proficient in providing adequate information 94.7% of the time. Microsoft Copilot showed the highest variability and the least efficacy. The artificial intelligence tools were generally proficient in responding to OMFS queries, with a collective adequacy rating of 74.5%, which increased to 84.6% when excluding Microsoft Copilot.

Conclusions: The study underscores the potential of AI chatbots, particularly ChatGPT-4, in providing reliable information for OMFS-related patient inquiries. While all tools showed proficiency to varying degrees, ChatGPT-4 demonstrated superior performance in both accuracy and consistency. The findings highlight the growing relevance of artificial intelligence in healthcare, suggesting a significant role for artificial intelligence chatbots in patient education and query resolution in OMFS. However, the variability among different tools indicates the need for careful selection and ongoing evaluation of artificial intelligence resources in clinical settings. At the present time we recommend oral surgeons only recommend the use of ChatGPT-4 for their patients. -van Leeuwen KG, de Rooij M, Schalekamp S, van Ginneken B, Rutten MJCM. How does artificial intelligence in radiology improve efficiency and health outcomes? Pediatr Radiol. 2022;52(11):2087-2093. doi:10.1007/s00247-021-05114-8 -Chow JCL, Sanders L, Li K. Impact of ChatGPT on medical chatbots as a disruptive technology. Frontiers in Artificial Intelligence. 2023;6. https://www.frontiersin.org/articles/10.3389/frai.2023.1166014

Figure 1
Figure 2
Figure 3