A medical professor pitted AI models against his students to see who would score better on his tests. The results were shocking!

In an ever-evolving world driven by technology, artificial intelligence (AI) has been making waves in nearly every sector. Workers are constantly threatened by the prospect of “AI taking their jobs”. But what if we told you that the tool could just as easily dominate our classrooms as well?

William Hersh, M.D., a seasoned professor at Oregon Health & Science University (OHSU), was particularly intrigued by AI's growing influence in his field. As a medical informatics educator, he sought to explore just how well generative AI could compete with his human students. The results? Surprising, to say the least.

An experiment in education

Hersh’s curiosity led him to conduct an experiment where he pitted six large-language AI models — including ChatGPT — against students enrolled in his introductory course on biomedical and health informatics. The goal was to see how AI would perform when tasked with the same assessments as his students. The findings, published in npj Digital Medicine, showed that AI models outperformed up to 75% of the class.

"This does raise concern about cheating, but there is a larger issue here," Hersh commented. "How do we know that our students are actually learning and mastering the knowledge and skills they need for their future professional work?"

The role of AI in knowledge-based learning

The study analysed the performance of 139 students who took the course in 2023, with AI models tackling multiple-choice questions and short-answer prompts. The AI models scored impressively, landing in the top 50th to 75th percentile. According to Hersh, these results raise essential questions about the future of student assessment, particularly in knowledge-heavy subjects like biomedical informatics.

Although the AI excelled in this particular course, Hersh was quick to point out that generative models might not fare as well in more hands-on or participatory academic settings, where critical thinking and complex skills development are vital. For Hersh, this balance between mastering knowledge and applying it in real-world scenarios is at the heart of medical education.

A new era for medical students

The AI's performance has stirred up an age-old debate in the academic community: how much information should students be expected to memorise versus being able to access through technological tools? Hersh recalls a time in medical school when one of his attending physicians insisted that every doctor should carry all medical knowledge "in their head." Even back then, Hersh saw the limitations of such an approach.

"The knowledge base of medicine has long surpassed the capacity of the human brain to memorise it all," he noted, emphasising the importance of making smart use of available technology.

Striking the right balance

Despite the impressive capabilities of AI, Hersh remains confident that medicine — and most academic disciplines — will always require a "human touch." Critical decision-making, nuanced judgement calls, and compassionate patient care cannot be fully replaced by machines.

"There are a lot of things that healthcare professionals do that are pretty straightforward, but there are those instances where it gets more complicated and you have to make judgement calls," Hersh said. "That's when it helps to have that broader perspective."

As fall classes approach, Hersh isn’t worried about AI encouraging cheating. His solution? Continuously updating the curriculum and ensuring that assessments go beyond what AI can quickly retrieve. "In any scientific field, there are new advancements all the time," he remarked. "Large-language models aren’t necessarily up to date on all of it."

Hersh’s experiment underscores the broader implications of AI in education. While AI has the potential to assist in areas such as medical knowledge recall, it cannot — and should not — replace the critical thinking, emotional intelligence, and human interaction that healthcare professionals need to succeed.

As AI continues to evolve, so too will educational approaches, aiming to equip students with both a solid foundation of knowledge and the tools to navigate an increasingly AI-enhanced world.