Residents spend a significant portion of their post-graduate training on-call. Traditionally, trainees are granted the highest levels of independence and autonomy during the after-hours on-call period, as supervising staff physicians are often not present in the hospital overnight. These experiences can be beneficial to learning and can allow for increased trainee confidence, readiness for independent practice, and the honing of clinical decision-making skills.1,2 Pedagogically, it would be useful for training programs to optimize learning during this period.

A recommended strategy to improve resident learning in any clinical setting is to provide feedback through formative assessment, such as a workplace-based assessment (WBA).3,4 WBAs are considered the cornerstone of competency-based medical education and are used as a proxy for what will be done in independent practice.5 There are a multitude of WBAs used in postgraduate training; however, few are designed specifically for the on-call context. Furthermore, there is a dearth of identified competencies specific to the on-call period.6

We identified a single study that described the development of an on-call assessment tool for ophthalmology residents.7 However, this study did not provide validity evidence based on modern validity theory8,9 and was not developed using established assessment-development principles.10,11 Other common WBAs that are routinely used may not apply during the on-call period as they rely on direct observation or are specifically designed for assessing technical skills or skills in simulation settings.12, 13, 14, 15

The success of competency-based medical education depends on regular assessment of resident competence across all care settings, and in order to optimize the educational impact of autonomous or semi-autonomous practice by trainees on-call, it is necessary to have appropriate, high-quality assessment tools available.1 Therefore, the objective of this study was to develop a formative, workplace-based assessment, for use in a competency-based surgical residency program to provide feedback to residents regarding on-call performance and collect validity evidence for its use.