Despite the increasing integration of POCUS within UME, there is a relative paucity of UME POCUS assessment tools that target the highest level of Miller’s pyramid reported in the literature. While assessing lower levels of Miller’s pyramid provides the advantage of ease of evaluation through knowledge-based MCQs and short answers, assessing higher levels of Miller’s pyramid enables more effective assessment of a learner’s competence in their day-to-day clinical work. A recent survey of UME directors demonstrated that the incorporation of questions into course examinations was the most common method of POCUS assessment [22].

Clinical assessment of learned skills allows for multiple subcompetencies of POCUS to be assessed including knowledge, identifying sonographic indications, demonstration of sonographic skills, image interpretation, and medical decision-making [8]. In an article by Olszynski et al. the authors successfully assessed the highest level of Miller’s pyramid in a clinical ultrasonography clerkship elective. Assessment methods were longitudinal and included multiple-choice examinations, technical skill examinations, and clinical assessment forms that were completed by clinical rotation supervisors. The goal of these clinical assessment forms was to assess the appropriateness and reliability of students’ skills in daily clinical practice. In an article by Krause et al. the authors assessed level four of Miller’s pyramid through a daily clinical assessment method in which students were required to complete and record a minimum of three clinically indicated extended Focused Assessment with Sonography in Trauma (eFAST) examinations during their surgical clerkship rotation [21]. An emergency staff physician or resident would then review the learner’s POCUS image and interpretation. At the same time, both of these authors also successfully integrated additional levels of Miller’s pyramid using knowledge-based examinations and technical skill assessments [18, 21]. Notably, one article by Andersen et al. provided limited training on handheld ultrasound devices to students then asked learners to acquire and interpret ultrasound images during their clinical rotations [20]. The images and interpretations were subsequently reviewed by staff physicians. This study demonstrated students ability to acquire and interpret their POCUS images in daily clinical practice with significant accuracy. The integration of handheld ultrasound devices and recording of images would allow a feasible assessment method to inform workplace-based assessments. Clinical indication, interpretation and clinical integration of POCUS images would need to be included in the assessment to provide a more robust evaluation of POCUS use in the workplace. The handheld ultrasound devices have the added advantage of increased accessibility and limited associated costs for UME programs [23].

A challenge associated with targeting Miller’s highest level of clinical assessment is the requirement for access to clinical environments. Due to the differences in medical school training and curricula across North America and even internationally, it may be difficult for pre-clinical learners to gain clinical opportunities prior to their formal clinical training. For these reasons, targeting level one, two, and/or three of Miller’s pyramid in the preclinical years, may be advantageous. The most common assessment method reported in the present scoping review for all articles was evaluation of technical performance. This included standardized assessments such as OSCEs, OSAUS, B-QUIET, and non-standardized tools which involved assessment of POCUS image acquisition skills. POCUS is a user-dependent skill and therefore acquisition and assessment of technical competence is an important component of competency. Standardized assessments such as OSCEs are beneficial in that they provide realistic simulations of patient care in a controlled environment. However, disadvantages associated with OSCE-style assessment methods include cost, time, and reliability of assessments across multiple stations [8]. If not successfully standardized, OSCEs are subject to observer bias and inter-rater agreement [6, 24]. Notably, one article in this review focused on transvaginal ultrasound training and used OSAUS as an objective assessment method while also assigning a global rating scale (GRS) using a five-point Likert scale [25]. While OSAUS provides an objective means of assessment, validity evidence has not yet been collected in the undergraduate medical student population [26].

Ultimately, employing a mixture of assessment methods that correspond to multiple levels of Miller’s pyramid may be the best approach to ensure a feasible and more comprehensive assessment of learned skills [10]. Slightly less than half of the articles from this scoping review used a multi-assessment approach integrating more than one level of Miller’s framework. The most common combination of assessment methods was evaluation of knowledge using MCQs and/or written examinations and evaluation of skills with technical demonstration. Because ultrasound clinical competency is multidimensional, educational models that assess for different subcompetencies are needed in UME. One example of such a model is the I-AIM tool, which stands for ‘indication, acquisition, interpretation, and medical decision making’ [27]. I-AIM is a standardized checklist for assessment of physician-performed focused sonographic examinations. Notably, one article in this scoping review introduced students to the I-AIM technique; however, the learned skills were assessed with written pre and post-knowledge tests rather than direct observation [28]. While the I-AIM model incorporates knowledge, technical skill, and medical decision making of ultrasonography, validity evidence for its use in undergraduate medical students is lacking [27]. The Ultrasound Competency Assessment Tool (UCAT) is another model that integrates multiple levels of Miller’s pyramid into POCUS assessment [29]. The UCAT consists of five domains including preparation, image acquisition, image optimization, clinical integration, and entrustment [29]. While not yet evaluated in the UME population, there is early validity evidence in POCUS competence for post-graduate Emergency Medicine trainees [29].

The future of assessing POCUS competence may benefit from a programmatic assessment approach that includes multiple levels of Miller’s pyramid using standardized and non-standardized methods. These methods can be formative assessments for the learner and then collected and analyzed by a faculty or committee to develop a rich diagnostic picture to allow a defensible, high-stakes decision of POCUS competence.