The performance of artificial intelligence language models in board-style dental knowledge assessment: A preliminary study on ChatGPT

Document Type

Article

Publication Date

9-5-2023

Publication Title

Journal of the American Dental Association

Keywords

Artificial intelligence, ChatGPT, dental board examination, dental education, dentistry, Integrated National Board Dental Examination

ISSN

0002-8177

Abstract

BACKGROUND: Although Chat Generative Pre-trained Transformer (ChatGPT) (OpenAI) may be an appealing educational resource for students, the chatbot responses can be subject to misinformation. This study was designed to evaluate the performance of ChatGPT on a board-style multiple-choice dental knowledge assessment to gauge its capacity to output accurate dental content and in turn the risk of misinformation associated with use of the chatbot as an educational resource by dental students.

METHODS: ChatGPT3.5 and ChatGPT4 were asked questions obtained from 3 different sources: INBDE Bootcamp, ITDOnline, and a list of board-style questions provided by the Joint Commission on National Dental Examinations. Image-based questions were excluded, as ChatGPT only takes text-based inputs. The mean performance across 3 trials was reported for each model.

RESULTS: ChatGPT3.5 and ChatGPT4 answered 61.3% and 76.9% of the questions correctly on average, respectively. A 2-tailed t test was used to compare 2 independent sample means, and a 2-tailed χ

CONCLUSION: ChatGPT3.5 did not perform sufficiently well on the board-style knowledge assessment. ChatGPT4, however, displayed a competent ability to output accurate dental content. Future research should evaluate the proficiency of emerging models of ChatGPT in dentistry to assess its evolving role in dental education.

PRACTICAL IMPLICATIONS: Although ChatGPT showed an impressive ability to output accurate dental content, our findings should encourage dental students to incorporate ChatGPT to supplement their existing learning program instead of using it as their primary learning resource.

DOI

10.1016/j.adaj.2023.07.016

This document is currently not available here.

Peer Reviewed

Find in your library

Share

COinS