With this study, we demonstrated the feasibility of implementing a framework for developing and delivering a 3-day AI curriculum in a radiology residency program. The framework consisted of five key steps (composing an AI expert team, assessing knowledge and needs of residents, defining audience and learning goals, determining educational format, and staring and evaluating a pilot) that other radiology departments may find useful to prepare a similar fast-track AI curriculum. The post-curriculum survey indicated that the curriculum improved participants’ self-reported confidence in how to handle AI-based approaches in radiology practice. Participants also recommended that the curriculum should be included as a standard component in the existing radiology residency program. Lectures on the fundamentals of AI and group discussions with experts were deemed most useful, while hands-on laboratory sessions with building an algorithm were rated as less useful. Furthermore, the participants perceived a course length of 3 days as sufficient. These results serve as a marker of the curriculum’s effectiveness, underlining its practical utility within radiology residency programs.

Two previous studies [12, 13] reported on the development and implementation of AI curriculum in radiology residency programs. Shiang et al. [12] evaluated residents’ real-time experience and perception of using AI-based decision support system applications. The residents found the approach desirable and reported positive experiences. However, a major limitation of their study was the lack of generalizability because they focused on the use of one commercially available platform rather than the concepts underlying AI-based systems and their practical application. Thus, this approach may not provide a thorough overview of the potential role of AI in radiology practice, and may not be applicable or appropriate for other institutions that use different platforms or vendors. Moreover, this study did not include a detailed description of learning goals on more foundational concepts of machine learning or deep learning. Nonetheless, we do agree that their design of training will prepare the participants for future AI advancements, and a real-time experience could be a part of a more comprehensive curriculum.

Hu et al. [13] developed a 3-week AI workshop. Their results on their post-workshop surveys showed increasing confidence in understanding AI concepts by the participating residents, similar to ours. They presented a comprehensive overview of methodology with learning objectives that make it more generalizable to other institutions. Nonetheless, a major limitation of this study is the length of the course (3 weeks), which limits the feasibility of embedding this course to radiology resident training programs. In our study, we designed a curriculum of 3 days, which the participants perceived as sufficient and the radiology residency program directors found feasible for easy incorporation into the existing radiology residency schedule. However, Hu et al. delved extensively into the technical aspects of AI, whereas we focused on broader aspects such as implementation, legislation, and ethics.

Recently, Salastekar et al. [15] highlighted the need for education in AI based on a survey among 759 residents in the USA. They found that a majority of radiology residents believed that education in AI should be included in the radiology training program. They found that hands-on laboratory sessions and didactic lectures were rated as the most effective learning methods but, in our study, especially the hands-on laboratory sessions, were not evaluated as most valuable. This may be because we used an open-source, research data set (Orange, https://orangedatamining.com). Some residents found this approach to be too technical (writing and adjusting algorithms), and preferred an easier and more visual method, which might be closer to the work of a radiologist. This is in line with the findings in other previous studies were residents felt AI to be important and worth learning, but most were not very interested in learning to program [16] or simplified, and self-contained coding environments could also serve as fertile opportunity for self-exploration [17]. However, preferences may not be the sole reasoning behind avoiding the technical intricacies. The team of AI experts still find the relative in-depth technical approach to be necessary. In future courses, balancing these perspectives, a less technical and a more visual approach, could be considered based on residents’ needs and expert recommendations.

Regarding the content, in our post-curriculum survey, participants felt that some topics were missing from the course. For example, input from radiologists currently working with AI, hospital management’ views on legal and insurance issues related to AI, and cost-effectiveness strategies of AI-powered tools in health care in Europe were not covered in our curriculum. Similar findings were reported by Huisman et al. [18] who surveyed 1041 radiologists on AI in radiology and concluded that AI education should include issues related to data management, ethics, and legislation. We addressed data management and implementation challenges by inviting a representative from a commercial vendor to provide insight during group discussions. This part of the curriculum can be further improved by including more (or specific) commercial AI software (Van Leeuwen et al. provided a full list of 100 commercially available AI software for radiology [19]), although this would add to the overhead by increasing the need for external resources and time commitment.

Ethical issues were also addressed during our sessions in the form of group discussions. However, we did not include an ethics expert with experience in clinical use of AI, and did not cover legislative aspects, hospital management views, or cost-effectiveness strategies in Europe. As these are emerging topics, it would be valuable to include group discussions with hospital management, policy makers, and insurance companies to discuss future challenges.

In our curriculum, we did not use any existing learning or vendor-based platforms. While some learning platforms are open source and can facilitate learning by providing access to the latest AI tools and resources, others come with a price tag. Depending on the budget and learning goals, the use of existing learning platforms could be useful while also allowing each institution to customize learning goals according to their own specific needs and practice.

Our study did not issue certificates upon course completion due to our non-accreditation status and the current lack of standardized certification for AI in healthcare. Recognizing the involvement of global entities in standardizing AI in healthcare, we suggest considering accreditation and skills evaluation as a next step for AI courses in the medical field. A recent study identified six competencies for physicians using AI tools in healthcare: foundational knowledge, critical appraisal, medical decision-making, technical use, patient communication, and awareness of unintended consequences [20]. A framework like this one could be used in shaping assessments for AI education of physicians.

This proposal presents some limitations that should be discussed. First, our findings are somewhat context dependent as different settings may have different needs and resources available. We compiled a team of AI experts, which may not be feasible in other institutions. Nonetheless, our institution does not have access to large platforms that are integrated into daily radiology practice yet, which may in fact be comparable to many institutions worldwide. Second, our self-reported pre- and post-assessment surveys may introduce bias by inflating participants’ confidence post-curriculum due to the positive experience of completing the curriculum. The absence of objective assessments to measure participants’ proficiency in understanding and evaluating AI software before and after the course is a limitation. To enhance future curricula, we recommend integrating more objective assessments pre- and post-curriculum, such as multiple-choice questions, case-based assessments, or practical exams, ensuring a more comprehensive evaluation of participants’ AI-related competence. Besides, in our surveys, we employed the Likert scaling method to evaluate various components of the curriculum. Additionally, integrating the net promoter score (NPS) may be beneficial, especially for queries related to recommending the curriculum. Third, our curriculum was developed and executed in a single academic hospital and with a small sample size (12 participants). Therefore, our results may not fully generalize to other radiology departments. However, its independence from a commercially available platform and its practical feasibility in terms of course duration would make it easier to implement in any residency program. Lastly, the implementation of the curriculum in a singular manner, combined with the reliance on a one-time post-curriculum survey, lacks a longitudinal assessment. Integrating an annually repeating curriculum accompanied by consistent survey results and experiential feedback would enhance its value. We plan to further personalize our AI curriculum to make it available annually to our radiology residents and to use annual survey’ results to further meet the needs of the participants continuously. Investigating these aspects further through a multicenter approach for future AI educational programs would be interesting. This could potentially offer a broader perspective and deeper insight into the perceptions and experiences of residents across different institutions.

In conclusion, designing an AI curriculum for radiology residents and implementing it into a radiology residency program is feasible using the framework presented. The 3-day AI curriculum effectively increased participants’ perception of knowledge and skills about AI in radiology and can serve as a starting point for further customization.