SA2 - Development of an Artificial Intelligence Algorithm in the Selection of Interviewees Among Applicants to Oral and Maxillofacial Surgery

Abstract: Study Objective The selection of candidates for interview among applicants to oral and maxillofacial surgery training programs is a consequential task for the applicant, program, and specialty as a whole. In addition to being time intensive, the process of selecting candidates for interview is fraught with the influence of conscious and unconscious bias. Furthermore, human decision making confers the inevitable elements of inter- and intra-reviewer variability and thereby subjects applicants to inconsistent selection standards. The utilization of an artificial intelligence algorithm in the selection of interviewees to U.S. allopathic medical schools has been demonstrated (1). Machine learning algorithms have also been developed for the selection of applicants to an internal medicine training program (2). To date, neither the development nor the use of a machine learning algorithm for the selection of interviewees in oral and maxillofacial surgery has been described. The objective of this study is to demonstrate the use of a machine learning algorithm in the selection of interviewees among applicants to a U.S. oral and maxillofacial surgery training program. Materials and Methods / Data Analysis Applicant data to the Northwell Health Oral and Maxillofacial Surgery training program was accrued from three consecutive application cycles (2021, 2022, 2023). The data was de-identified such that neither an applicant’s identity nor ethnicity could be determined from the data sets. Sixty percent of the collective data was used as a training set (m=308), while twenty percent was used for validation (m=102) and twenty percent used as a test set (m=102). A machine learning model was constructed consisting of data preprocessing, feature generation, and predictive modeling. The code was written in Python 3.7 (https://www.python.org/downloads/release/python-370/) and predictive modeling was performed using XGBoost18 (https://xgboost.readthedocs.io/en/latest/), an ensemble tree-based learner with gradient boosting. Hyperparameter tuning was optimized on the validation set using binary cross-entropy and F1 scores. Accuracy and precision data were obtained to assess accuracy and precision for training, validation and test sets. Congruence with program leadership selection of interviewees was assessed with the test set. Results and Outcomes Data Preliminary training set data demonstrated an accuracy of 90% with a precision of 93% (F1 = 0.83). Validation set data demonstrated an accuracy of 79% with precision of 69% (F1=0.63). The accuracy of the test set was measured at 79% with a precision of 58% (F1=0.56). CBSE score was the parameter with the highest F score for feature importance (fig. 1) and demonstrated the highest impact on model output (fig. 2). Conclusions The potential to reduce bias and enhance consistency in the selection of candidates remains a fundamental advantage of using machine learning for this purpose. While preliminary data is promising, a significantly higher quantity of data points is required to optimize the accuracy and precision of test set selection. Synthetic data sets are currently being generated which will be used to further train the algorithm. Future analysis of accuracy and precision data may be performed on new application cycles to further characterize the utility of machine learning for the selection of interviewees.

References: 1. Graham Keir, Willie Hu, Christopher G Filippi, Lisa Ellenbogen, Rona Woldenberg, Using artificial intelligence in medical school admissions screening to decrease inter- and intra-observer variability, JAMIA Open, Volume 6, Issue 1, April 2023, ooad011, https://doi.org/10.1093/jamiaopen/ooad0112. Burk-Rafel, Jesse MD, MRes1; Reinstein, Ilan MS2; Feng, James MD, MS3; Kim, Moosun Brad MS4; Miller, Louis H. MD5; Cocks, Patrick M. MD6; Marin, Marina MSc7; Aphinyanaphongs, Yindalon MD, PhD8. Development and Validation of a Machine Learning-Based Decision Support Tool for Residency Applicant Screening and Review. Academic Medicine 96(11S):p S54-S61, November 2021. | DOI: 10.1097/ACM.0000000000004317

Feature Importance by F Score
Parameter Impact on Model Output