(P101) ECHOING INTELLIGENCE: ASSESSING THE CAPABILITIES OF PODCASTAI (A NOVEL GENERATIVE AI SYSTEM) IN CREATING AND TAKING ECHOCARDIOGRAPHY EXAMS

Thursday, October 24, 2024

1:30 PM – 2:00 PM PT

Location: BOARD 69

Presenting Author(s)

Muneeb Ahmed, MD

Internal Medicine Resident Physician
University of Toronto
Mississauga, Ontario, Canada

Author(s)

Muneeb Ahmed, MD

Internal Medicine Resident Physician
University of Toronto
Mississauga, Ontario, Canada

First Author(s)

Muneeb Ahmed, MD

Internal Medicine Resident Physician
University of Toronto
Mississauga, Ontario, Canada

Co-Author(s)

FH

Flora Huang, MD

Echocardiography Fellow
University of Toronto
Toronto, Ontario, Canada
EB

Edward Brawer, n/a

CEO, PodcastAI, Inc.
San Mateo, California, United States

Senior Author(s)

Chi-Ming Chow, MD MSc FRCPC FCCS FASE

Professor
St. Michael's Hospital
Toronto, Ontario, Canada

Supervisor(s)

Chi-Ming Chow, MD MSc FRCPC FCCS FASE

Professor
St. Michael's Hospital
Toronto, Ontario, Canada

Disclosure(s):

Muneeb Ahmed, MD: No financial relationships to disclose

Background: The advent of artificial intelligence (AI) in medical education has sparked interest in its capability to both generate relevant questions and provide accurate answers. This study aims to evaluate the effectiveness of Podcast AI in creating questions for cardiology educational content and its ability to answer questions in comparison with cardiology fellows.

METHODS AND RESULTS: A total of 10 lectures from the SMH Virtual Echo Rounds podcast were selected. For each lecture, one question was generated by Podcast AI and another by a human cardiologist, resulting in a pool of 20 questions. A comprehensive exam consisting of these questions was administered to cardiology fellows. Each question was followed by a query asking fellows to identify if it was AI-generated or human-generated. Additionally, Podcast AI's ability to correctly answer human-made questions was assessed and compared to the performance of cardiology fellows.

Preliminary analysis indicates similarities in both Podcast AI and eight cardiology trainees’ ability to identify AI-generated or human generated question (55% vs 52%). When comparing the accuracy of responses, Podcast AI scored 100% on human-generated questions and 95% on AI-generated questions while the fellows scored 74% on human-generated questions and 75% on AI-generated questions.

Conclusion: The results of this study suggest that Podcast AI is highly effective in generating cardiology educational content that is indistinguishable from questions generated by human cardiologists. The ability of both Podcast AI and cardiology fellows to identify the source of the questions with similar accuracy further supports the sophistication of the AI in creating relevant educational material. Moreover, Podcast AI's ability to correctly answer both AI-generated and human-generated questions with a higher accuracy rate than cardiology fellows highlights its potential as a valuable educational tool. However, the human touch in education, the nuanced understanding of complex topics, and the interactive learning experience that human educators provide are irreplaceable. The integration of AI in medical education should aim to complement and enhance these human elements rather than replace them. Moving forward, the use of AI like Podcast AI in cardiology education could be optimized to support fellows by providing additional practice questions and explanations, thereby enriching the learning experience and potentially improving educational outcomes.