Development of large language model-generated multiple-choice questions and flashcards to enhance the Canadian undergraduate urology curriculum
DOI:
https://doi.org/10.5489/cuaj.9845Keywords:
Artificial Intelligence, Education, Benign Prostatic Hyperplasia (BPH), UrolithiasisAbstract
Introduction: The Canadian Undergraduate Urology Curriculum (CanUUC) is a free, online curriculum that covers important topics in urology for medical students. While multiple-choice questions (MCQs) and flashcards are widely used for medical education, they can be resource-intensive and time-consuming to produce. Therefore, to strengthen the CanUUC with such tools, we developed a customized large language model (LLM)-enabled system to generate MCQs and flashcards from CanUUC material and evaluated the quality of its outputs.
Methods: We used OpenAI’s GPT-5 model and refined two system prompts to generate educational materials from two CanUUC topics (benign prostatic hyperplasia and urolithiasis). Six evaluators blinded to items’ origin were involved, and each item received two independent ratings for relevance, correctness, clarity, distractor quality, and suitability for education. Items with a mean score <3 on the suitability for education evaluation domain were excluded. Results were summarized descriptively.
Results: Ninety items, including 40 MCQs and 50 flashcards were generated. Among flashcards, 22% allowed >1 plausible answer. Across all independent ratings, the proportion of top scores (4–5) ranged from 85.5−90.5% across all domains. Overall, 86.1% of independent ratings judged items suitable for medical education. When averaging the suitability score at the item-level, 12 items (13%) had a score <3 and were excluded. Mandatory (8%) and optional (27%) adjustments were suggested to refine items’ clarity and correctness. From the initially generated items, 87% were included in the CanUUC.
Conclusions: Our customized system can produce curriculum-aligned MCQs and flashcards that meet item writing standards and are mostly suitable for medical education. Expert review is essential to identify flaws and improve quality before inclusion into the curriculum.
Downloads
Downloads
Published
How to Cite
Issue
Section
License
You, the Author(s), assign your copyright in and to the Article to the Canadian Urological Association. This means that you may not, without the prior written permission of the CUA:
- Post the Article on any Web site
- Translate or authorize a translation of the Article
- Copy or otherwise reproduce the Article, in any format, beyond what is permitted under Canadian copyright law, or authorize others to do so
- Copy or otherwise reproduce portions of the Article, including tables and figures, beyond what is permitted under Canadian copyright law, or authorize others to do so.
The CUA encourages use for non-commercial educational purposes and will not unreasonably deny any such permission request.
You retain your moral rights in and to the Article. This means that the CUA may not assert its copyright in such a way that would negatively reflect on your reputation or your right to be associated with the Article.
The CUA also requires you to warrant the following:
- That you are the Author(s) and sole owner(s), that the Article is original and unpublished and that you have not previously assigned copyright or granted a licence to any other third party;
- That all individuals who have made a substantive contribution to the article are acknowledged;
- That the Article does not infringe any proprietary right of any third party and that you have received the permissions necessary to include the work of others in the Article; and
- That the Article does not libel or violate the privacy rights of any third party.
