Does ChatGPT reveal a real potential in medical education or is it generating more hype than substance?

ChatGPT is a new artificial intelligence (AI) system, known as a large language model (LLM), designed to generate human-like writing by predicting upcoming word sequences. Unlike most chatbots, ChatGPT cannot search the internet. Instead, it generates text using word relationships predicted by its internal processes.

According to a study published this month, researchers Tiffany Kung, Victor Tseng and colleagues at AnsibleHealth tested ChatGPT’s performance on the United States Medical Licensing Exam (USMLE), a highly standardised and regulated series of three exams taken by medical students and physicians-in-training.

ChatGPT scored between 52.4% and 75.0% across the three USMLE exams. The passing threshold each year is approximately 60%. ChatGPT also demonstrated 94.6% concordance across all its responses and produced at least one significant insight for 88.9% of its responses.

Alex Polyakov, Associate Professor in the Department of Medical Education, the University of Melbourne Medical School, said “The results highlight the potential of AI-based systems like ChatGPT to enhance medical education, specifically in areas such as student assessment, knowledge dissemination, and curriculum development. AI-based systems like ChatGPT can provide quick and efficient feedback to medical students and physicians-in-training, allowing them to identify areas for improvement and further study. Additionally, ChatGPT’s ability to simplify and synthesise information could make it a valuable tool in disseminating knowledge to medical students, especially in the early stages of their education.

“Moreover, ChatGPT’s ability to provide insights and counterpoints could also be useful in developing new medical curricula. AI-based systems like ChatGPT can provide a wealth of information and ideas, which can help medical educators create innovative and engaging learning experiences for medical students.

“However, it is important to note that while the results are promising, further research is needed to evaluate the long-term impact and reliability of AI-based systems like ChatGPT in medical education. Additionally, AI-based systems like ChatGPT should not replace human interaction and assessment in medical education. Medical students and physicians-in-training need to develop critical thinking skills, clinical reasoning, and ethical awareness, all of which require human interaction and feedback.”

Dr Simon McCallum, Senior Lecturer in Software Engineering, Te Heranga Waka, Victoria University of Wellington, stated “Within university education we are having to pivot almost as fast as at the start of the pandemic to account for the ability of AI to perform tasks which were traditionally a sign of understanding. There is also a massive cultural shift when everybody has access to a tool that can assist in written communication. Careers and jobs which were seen as difficult, may be automated by these AI tools. Microsoft has announced that ChatGPT is now integrated into MS Team Professional and will act as a meeting secretary, summarising meetings and creating action items. Bing will also include a ChatGPT advancement linking the version 4 of ChatGPT with up-to-date search information.

“Society is about to change, and instead of warning about the hypochondria of randomly searching the internet for symptoms, we may soon get our medical advice from Doctor Google or Nurse Bing.”

Dr Collin Bjork, Senior Lecturer in Science Communication and Podcasting, Massey University, shared an opposing perspective on these study results.

He said “The claim that ChatGPT can pass US medical exams is overblown and should come with a lengthy series of asterisks. Like ChatGPT itself, this research article is a dog and pony show designed to generate more hype than substance.

“OpenAI had much to gain by releasing a free open-access version of ChatGPT in late 2022 and fomenting a media fervour around the world. Now, OpenAI is predicting 1 billion in revenue in 2024, even as a ‘capped-profit’ company.

“Similarly, the authors of this article have much to gain by releasing a free open-access version of their article claiming that ChatGPT can pass the US Medical Licensing Exams. All of the authors but one work for Ansible Health, ‘an early stage venture-backed healthcare startup’ based in the Silicon Valley. At two years old, this tiny company will likely need to go back to their venture capitalist investors soon to ask for more money. And the media splash from this well-timed journal article will certainly help fund their next round of growth. After all, a pre-print of this article already went viral on social media because the researchers listed ChatGPT as an author. But the removal of ChatGPT from the list of authors in the final article indicates that this too was just a publicity stunt.

“As for the article itself, the findings are not as straightforward as indicated. Here’s one example:

“The authors claim that ‘ChatGPT produced at least one significant insight in 88.9% of all responses’. But their definition of ‘insight’ as ‘novelty, nonobviousness, and validity’ is too vague to be useful. Furthermore, the authors insist that these ‘insights’ indicate that ChatGPT ‘possesses the partial ability to teach medicine by surfacing novel and nonobvious concepts that may not be in the learner’s sphere of awareness’. But how can an unaware learner distinguish between true and false insights, especially when ChatGPT only offers ‘accurate’ answers on the USMLE a little more than half the time?

“The authors’ claims of ChatGPT’s insights and teaching potential are misleading and naive.”

In reimagining healthcare, Health Industry Hub^TM is the ONLY one-stop-hub uniting the diversity of Pharma, MedTech, Diagnostics & Biotech sectors to inspire meaningful change. The exclusive leadership and influencer podcasts and vodcasts offer unparalleled insights and add immense value to our breaking news coverage.

The Health Industry Hub^TM content is copyright protected. Access is available under individual user licenses. Please click here to subscribe and visit T&Cs here.