In 2007, the Chilean Society of Orthopedics and Traumatology (SCHOT) began the process of developing a multiple-choice examination for all residents who graduated from orthopaedic university programs in Chile (National Medical Graduates [NMG]).1 Similar examinations exist in countries such as Spain, Brazil, the United States, Canada, and Asian countries.2–7 The National Medical Examination of Orthopaedics and Traumatology (EMNOT) was first implemented in 2009 and designed as a formative test to diagnose the academic performance of graduate students from different programs. Since then, it has provided confidential results to Chilean universities to detect areas for improvement in their academic programs. Given its formative role, it was established as a voluntary examination and lacked a certifying role for Chilean graduates.1 In 2014, the Chilean government implemented a plan to boost the number of graduates from all medical specialties to meet the ever-increasing demands for specialists in public health services.8 This push for more specialists increased the number of orthopaedic surgeons graduating yearly. Simultaneously, there was a notable migratory flow of other Latin American orthopaedic surgeons moving to practice in Chile.9

The National Medical Specialties Corporation (International Medical Graduates [IMG]), in charge of the revalidation of medical specialists, decided that the EMNOT developed by SCHOT would be part of the accreditation process for foreign orthopaedic surgeons who wanted to practice in Chile (title revalidation).10

Despite the educational effect it has on Chilean orthopaedic programs and its role as an accreditation examination, there are no published studies regarding psychometric characteristics of EMNOT. Therefore, the goal of this study was to describe participation and performance according to the examinee's origin and analyze the difficulty and discrimination indexes of the EMNOT during its 11 years of implementation (2009 to 2019).

Methods

We obtained institutional board approval from the President and Board of Directors of the SCHOT and the Scientific Ethics Committee of the Faculty of Medicine of the Pontificia Universidad Católica de Chile (ID 190925008). We retrospectively assessed all EMNOT results between 2009 and 2019. To ensure confidentiality, no identifying data of the universities or examinees were included. The EMNOT consists of 120 multiple-choice questions, according to a common theoretical framework agreed upon and defined by the SCHOT Teaching Committee. This content profile is publicly available and generated from analyzing the different national university programs accredited by the Accrediting Agency for Postgraduate Programs of Specialties in Medicine (APICE).11 The examination scores reflect a percentage of correct answers (zero to 100 points), with no deductions for incorrect answers. In 2012, SCHOT established that graduates with 65 or more points would be qualified to become an active member of the Society, and in 2014, IMG established this same grade for revalidating orthopaedic surgeons.10 Since 2015, given the increasing demand, the EMNOT has been administered twice a year.

The number of examinees, participating universities, and scores and percentage of examinees approved (score greater than or equal to 65 points) were recorded for the 2009 to 2019 period. In addition, the evolution of these variables was also analyzed.

We compared the results obtained between Chilean university program (NMG) graduates and those revalidating foreign orthopaedic surgeons (IMG). In addition, we analyzed the result of different Chilean university programs, comparing their results according to the number of accreditation years given by APICE (not accredited universities, 3, 5, or 7 years of accreditation).

Difficulty index (P) is the proportion of examinees who answered correctly, using the formula P = (R/Q), where R is the average percentage of correct answers and Q is the maximum examination score.

Discrimination index was calculated (D) using the formula D = (Smax − Smin)/Q, where Smax is the maximum score, Smin is the minimum score, and Q is the number of questions. The discrimination index is the instrument's ability to distinguish between high and low performing examinees.12–14 The interpretation of both indexes is presented in Tables 1 and 2.

Table 1 - Interpretation of Difficulty Index12,14 Range Difficulty Level ≥90 Unsuitable easy 80-89 Very easy 70-79 Easy 50-69 Good 40-49 Difficult 10-40 Very difficult ≤9 Unsuitable difficulty
Table 2 - Interpretation of Discrimination Index12,13 Range Discrimination Level ≥40 Excellent discrimination 30-39 Good discrimination 20-29 Fair discrimination ≤19 Poor discrimination

The distribution of the numerical variables was analyzed with the Shapiro-Wilkin test and summarized as means and SDs. Categorical variables were described using absolute and relative frequencies. For the inferential analysis, chi square test was used to evaluate associations of categorical variables and Student t-test to compare means of independent variables. Previously, we tested variance homogeneity with the Levene test.

Association between variables was measured using linear and logistic regression models. Significance was determined as a P value < 0.05. Statistical analysis was conducted with STATA 16 software (StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC).

Results

Between 2009 and 2019, a total of 975 examinees took the EMNOT, of whom 41.23% (402) were from a Chilean university training program (NMG).

Participating universities increased from 4 in 2009 to 17 in 2019, with an average increase of one university program per year (95% CI, 0.96 to 1.42; P < 0.001). NMG examinees increased an average of six per year (95% CI, 5.53 to 7.22; P < 0.001) while IMG examinees increased at an average rate of 35 examinees per year (95% CI, 19.55 to 50.56; P = 0.033) (Figure 1).

Figure 1:

Graph showing participation according to the number of examinees of the Chilean university programs (NMG) and IMG between 2009 and 2019. IMG = International Medical Graduates, NMG = national medical graduates

The average EMNOT score between 2009 and 2019 was 61.88 (±11.36). Regarding the origin of the examinees, NMG examinees had a significantly higher score than IMG with a mean score of 66.53 (±8.67) versus 51.30 (±11.42, P < 0.001), respectively (Figures 2 and 3A). The average scores obtained by the NMG and IMG examinees did not vary during the studied period (NMG P = 0.42; IMG P = 0.40) (Figure 2).

Figure 2:

Graph showing performance according to the average score of examinees of the Chilean university programs (NMG) and IMG between 2009 and 2019. IMG = International Medical Graduates, NMG = national medical graduates

Figure 3:

Graphs showing comparison of scores obtained in the EMNOT among examinees. A, Comparison between IMG and NMG examinees; B, comparison between IMG and NMG accredited and non-accredited examinees; C, comparison between accredited NMG according to years of APICE accreditation. APICE = Accrediting Agency for Postgraduate Programs of Specialties in Medicine, EMNOT = National Medical Examination of Orthopaedics and Traumatology, IMG = International Medical Graduates, NMG = national medical graduates

Among NMG examinees, we found a significant difference (P < 0.001) between the average scores of accredited programs (70.04 ± 7.87) when compared with those of programs that have not completed the accreditation process (64.63 ± 8.49) (Figure 3B).

A statistical difference was also seen considering the number of years for which each program was accredited. The average score was 63.07 ± 11.18 for those accredited for 3 years, 69.70 ± 7.00 for those with 5-year accreditation, and 72,23 ± 7.26 for the 7-year programs (Figure 3C). A statistical difference was seen between 3 and 5-year (P = 0.008), between 3 and 7-year (P < 0.001), and between 5 and 7-year programs (P = 0.038) (Figure 3C).

In the 2014 to 2019 period, the percentage of examinees who scored ≥ 65 points was 45.03% (353/803). When analyzed according to the origin of the examinees, there was a significant difference in favor of NMG examinees (63.97% [261/408]), when compared with IMG examinees (23.29% [92/395]) (P < 0.001). The probability of “approving” the examination (score ≥ 65 points) was 5.32 times higher in the NMG than IMG examinees (95% CI OR, 3.94 to 7.19; P < 0.001). No differences were observed in the percentage of examinees who scored ≥ 65 points regarding the year in which the examination was given (NMG P = 0.90; IMG P = 0.92).

When comparing the percentage of examinees who scored ≥65 points among graduates from accredited versus non-accredited NMG programs, we observed that 83.94% (115/137) of accredited program examinees while only 53.87% (146/271) of those from non-accredited programs achieved scores ≥65. The probability of achieving a score ≥65 in accredited programs was 3.84 times higher than in non-accredited programs (95% CI OR, 2.43 to 6.08; P < 0.001).

Differences were observed between number of years of accreditation and chances of approval (score ≥ 65 points). Examinees from 3-year programs approved 57.14%, in contrast to 88.10% (74/84) of the 5-year programs and 84.62% (33/39) in the 7-year ones. Compared with programs accredited for 3 years, those accredited for 5 years had a 3.64 times greater probability (95% CI, 1.24 to 10.67; P = 0.01) of approving and those with 7 years 1.98 times higher (95% CI, 1.12 to 3.51; P = 0.01). No differences were observed in the probability of a score ≥65 points between the 5 and 7-year programs (P = 0.94).

The average difficulty index for the 2009 to 2019 period was adequate (62.93 ± 5.07), varying between easy and good along the period (Figure 4). The average discrimination index was adequate (0.42 ± 0.94), varying between fair and excellent (Figure 5).

Figure 4:

Graph showing difficulty index of the EMNOT between 2009 and 2019. EMNOT = National Medical Examination of Orthopaedics and Traumatology

Figure 5:

Graph showing EMNOT discrimination index between 2009 and 2019. EMNOT = National Medical Examination of Orthopaedics and Traumatology

Discussion

EMNOT aims to measure residents' knowledge at the end of their training period, thus providing feedback to their universities to improve Chilean orthopaedic resident training process continually. Since 2014, EMNOT has also played a dual role as a revalidation examination in the process for foreign specialists who want to work as orthopaedic surgeons in our country.10

The latter affects the access of foreign specialists into the Chilean public health system, which is why the tests’ correct construction and application is even more relevant. We observed that the results of the EMNOT differ according to the training of the examinees and that during the studied period, the examination had adequate difficulty and discrimination indexes.15,16

Many countries use these tests at the end of the orthopaedics residency to assess residents and training programs.1,2,4–6 The EMNOT results since 2009 have remained continuous, despite the notable increase in the number of examinees and participating universities. The expansion of both program variety and the range of offerings within each program is notable, encompassing private university programs as well. This growth could be related to the pressure exerted by the Ministry of Health on universities to train an increasing number of specialists to meet the high public service demand.8,10,11,17 We also should consider the explosive growth of IMG examinees. The abrupt migratory flows from Latin American countries during the past 5 years9,18 could explain this, coupled with the government's decision to authorize revalidation of medical licenses for those professionals who pass their specialist certification examinations.10

The progressive rise of examinees in the country is similar to what happened in Europe with the European Board of Orthopaedics and Traumatology examination. This evaluation was implemented in 2010 and consists of a written and oral section. It is a requirement for European certified specialists and non-European specialists who want to be certified to work or study in Europe.2 Similarly, in the United States, the “Orthopaedics In-Training Examination” has been applied since 1963 as a non-qualifying training instrument,3,6 with a growth of 68% in those examined between 1963 and 1969.3,19 They also have a similar examination called American Board of Orthopaedic Surgery Part I,20 which is taken at the end of the 5-year residency to achieve certification as a specialist and to apply for membership in the American Academy of Orthopaedic Surgeons.21 Unlike the Orthopaedics In-Training Examination or the EMNOT, the American Board of Orthopaedic Surgery examination has a qualification and certification purpose, and the accreditation council uses its results to appraise the quality of the programs offered by different institutions.

These examinations are voluntary in some countries, such as Sweden and Spain. In the case of Spain, since 2005, this examination has been a “voluntary final evaluation for resident doctors of Orthopedic Surgery and Traumatology.” Like the EMNOT, it also has shown a progressive growth in its level of participation.2

In this study, distinct differences were observed in their performance according to the origin of the examinees. The NMG examinees obtained markedly higher scores than the IMG (66.52 versus 55.13, respectively). A similar result is observed in the percentage of approval, between 63.97% for the NMG examinees and 23.29% for the IMG examinees. This difference observed between national and foreign examinee performance has also been seen in the Chilean examination for revalidation of the medical degree.22 Several reasons could explain these results, starting with the difference in the training syllabus between Chilean and foreign universities, which varies according to the reality of each country. The same examinees have also stated that the EMNOT, designed to test final-year residents, includes topics that surgeons who are subspecialized or have practiced for several years have not reviewed in a long time, finding themselves at a disadvantage compared with recently graduated orthopaedic surgeons.

International migration of physicians has increased in recent decades.23 In recent decades, there has been a notable uptick in the international migration of physicians. Presently, a substantial portion of surgeons from low and middle-income nations are practicing in high-income countries.24 In this context, IMGs must adhere to the stipulations set forth by the medical registration authorities in each respective country to engage in medical practice. Orthopaedic surgeons are not exempt from this prevailing global phenomenon. Furthermore, when we take into account the variations in the content and duration of orthopaedic training programs, it becomes imperative to establish structured objective examinations for the purpose of certifying the knowledge of IMGs.23

Contrarily, within the NMG examinees, differences in performance were also observed between accredited programs and not accredited programs. Among the accredited programs, differences were also seen according to the years of accreditation and could be explained because accredited programs deliver better quality by meeting more of APICE requirements.25

Analysis of each EMNOT's question by calculating difficulty and discrimination indexes provides feedback on what the students have learned and enables instructors to determine and correct the deficient areas. They are part of the psychometric analysis of an examination and contribute to increasing the validity and reliability of the tests by revealing whether the items are working well. The results obtained for the studied period show adequate indices of difficulty and discrimination. In other words, these results show that most of the test questions were in an appropriate range of difficulty and that the test can distinguish between examinees’ knowledge.26

Regarding the limitations of this study, the EMNOT, like any multiple-choice examination, does not assess the practical skills required by any orthopaedic surgeons. In addition, this study analyzes the results of implementation of the EMNOT and the indexes of difficulty and discrimination. However, it does not include other sources of validity specific to the examination, such as content validity, construct validity, and the relationship with other variables.15 It would be necessary for additional studies to assess all these aspects.

Conclusion

The Chilean EMNOT serves as a notable illustration for a standardized and officially recognized examination for the purpose of certifying the knowledge of IMGs.

It has grown markedly in the number of examinees from 2009 to 2019, distinguishes according to the origin of the examinees, and shows adequate difficulty and discrimination indexes.

Acknowledgments

SCHOT (Chilean Society of Orthopaedic Surgeons).

References 1. Urrutia J, Orrego M, Wright AC, Amenabar D: An assessment of the Chilean national examination of orthopaedic surgery. BMC Med Educ 2016;16:78. 2. Urda A, Ezquerra L, Albareda J, et al.: Final voluntary assessment for traumatology and orthopaedic surgery medical residents: A report on the results and a look at the future [in Spanish]. Rev Esp Cir Ortop Traumatol 2012;56:188-196. 3. Mankin HJ: The orthopaedic in-training examination (OITE). Clin Orthop Relat Res 1971;75:108-116. 4. Marsh JL, Hruska L, Mevis H: An electronic orthopaedic in-training examination. J Am Acad Orthop Surg 2010;18:589-596. 5. Cho Y, Kim JY, Park JH: Analysis of the Korean orthopedic in-training examination: The hip and pelvis section. Hip Pelvis 2016;28:157-163. 6. Evaniew N, Holt G, Kreuger S, et al.: The orthopaedic in-training examination: Perspectives of program directors and residents from the United States and Canada. J Surg Educ 2013;70:528-536. 7. Camanho GL: SBOT will be responsible for basic orthopedic formation. Rev Bras Ortop 2017;52:123. 8. Ministerio De Salud: Plan de Especialistas Médicos 2014-2018, 2014. Available at: http://web.minsal.cl/sites/default/files/files/Plan.deespecialistas.pdf. Accessed October 17, 2023. 9. Iberoamericano O, Movilidad S, Guizardi L: Chile y la migración, 2016, pp 11. Available at: https://www.bcn.cl/siit/actualidadb…/chile-y-la-migracion-los-extranjeros-en-chile.Encaché. Accessed October 17, 2023. 10. Ministerio de Salud: Subsecretaría de Salud Pública. Ley 20261. Biblioteca del Congreso Nacional de Chile, 2008. Available at: https://www.bcn.cl/leychile/navegar?idNorma=270584. Accessed October 17, 2023. 11. APICE: Estado Acreditación Programas, 2020. Available at: http://www.apicechile.cl/site/proceso.php. Accessed October 17, 2023. 12. Backhoff E, Larrazolo N, Rosas M: The level of difficulty and discrimination power of the basic knowledge and skills examination (EXHCOBA). Rev Electrónica Investig Educ 2000;2:1-16. 13. Ebel RL, Frisbie DA: Essentials of Educational Measurement. Englewood Cliffs, NJ, Prentice Hall, 1972. 14. Tartarini E: Evaluación escolar y elementos de estadística aplicada. Santiago, Chile, 1967. 15. Downing SM: Validity: On meaningful interpretation of assessment data. Med Educ 2003;37:830-837. 16. Downing SM: Reliability: On the reproducibility of assessment data. Med Educ 2004;38:1006-1012. 17. Gobierno del Estado de Yucatán: Diario Oficial Números 2013:73. Available at: http://www.desarrollorural.yucatan.gob.mx/MarcoJur/LeyDRSEY.pdf. Accessed October 17, 2023. 18. Ministerio de Interior y Seguridad. Departamento de Extranjería y Migración. 2020. Available at: https://www.extranjeria.gob.cl/estadisticas-migratorias/. Accessed October 17, 2023. 19. Nattress L: The role of the university in graduate medical education. 3. Orthopaedic in-training examination. J Med Educ 1969;44:878-889. 20. DeRosa GP: Recertification: History of ABOS efforts. Clin Orthop Relat Res 2006;449:149-154. 21. AAOS: About ABOS, 2020. Available at: https://www.aaos.org/about/. Accessed October 17, 2023. 22. Ministerio de Salud Chile: Evaluación del examen único nacional de conocimientos de medicina de Chile, 2018. Available at: http://biblioteca.digital.gob.cl/handle/123456789/3629. Accessed October 17, 2023. 23. Hohmann E, Tetsworth K: Fellowship exit examination in orthopaedic surgery in the commonwealth countries of Australia, UK, South Africa and Canada. Are they comparable and equivalent? A perspective on the requirements for medical migration. Med Educ Online 2018;23:1537429. 24. Lantz A, Holmer H, Finlayson SRG, et al.: Measuring the migration of surgical specialists. Surgery 2020;168:550-557. 25. APICE: Requisitos Específicos de un Programa de Formación de Especialistas en Programa de Traumatología y Ortopedia, 2014. Available at: http://www.apicechile.cl/images/stories/doc/imagenes/traumatologia_ortopedia.pdf. Accessed October 17, 2023. 26. Koçdar S, Karadağ N, Şahin: Analysis of the difficulty and discrimination indices of multiple-choice questions according to cognitive levels in an open and distance learning context. Turkish Online J Educ Technol 2016;15:16-24.