Setting proficiency standards for simulation-based mastery learning of short antegrade femoral nail osteosynthesis: a multicenter study
DOI:
https://doi.org/10.2340/17453674.2024.40812Keywords:
Assessment, Education, Fractures, Hip, Mastery learning, Mastery standard, Simulation, TrainingAbstract
Background and purpose: Orthopedic trainees frequently perform short antegrade femoral nail osteosynthesis of trochanteric fractures, but virtual reality simulation-based training (SBT) with haptic feedback has been unavailable. We explored a novel simulator, with the aim of gathering validity evidence for an embedded test and setting a credible pass/fail standard allowing trainees to practice to proficiency.
Patients and methods: The research, conducted from May to September 2020 across 3 Danish simulation centers, utilized the Swemac TraumaVision simulator for short antegrade femoral nail osteosynthesis. The validation process adhered to Messick’s framework, covering all 5 sources of validity evidence. Participants included novice groups, categorized by training to plateau (n = 14) or to mastery (n = 10), and experts (n = 9), focusing on their performance metrics and training duration.
Results: The novices in the plateau group and experts had hands-on training for 77 (95% confidence interval [CI] 59–95) and 52 (CI 36–69) minutes while the plateau test score, defined as the average of the last 4 scores, was 75% (CI 65–86) and 96% (CI 94–98) respectively. The pass/fail standard was established at the average expert plateau test score of 96%. All novices in the mastery group could meet this standard and interestingly without increased hands-on training time (65 [CI 46–84] minutes).
Conclusion: Our study provides supporting validity evidence from all sources of Messick’s framework for a simulation-based test in short antegrade nail osteosynthesis of intertrochanteric hip fracture and establishes a defensible pass/fail standard for mastery learning of SBT. Novices who practiced using mastery learning were able to reach the pre-defined pass/fail standard and outperformed novices without a set goal for external motivation.
Downloads
References
Palm H, Jacobsen S, Krasheninnikoff M, Foss, N B, Kehlet H, Gebuhr P. Influence of surgeon’s experience and supervision on re-operation rate after hip fracture surgery. Injury 2007; 38: 775-9. doi: 10.1016/j.injury.2006.07.043. DOI: https://doi.org/10.1016/j.injury.2006.07.043
Authen A L, Dybvi, E, Furnes O, Gjertsen J. Surgeon’s experience level and risk of reoperation after hip fracture surgery: an observational study on 30,945 patients in the Norwegian Hip Fracture Register 2011–2015. Acta Orthop. 2018; 89(5): 496-502. doi: 10.1080/17453674.2018.1481588. DOI: https://doi.org/10.1080/17453674.2018.1481588
Cook D A, Brydges R, Hamstra S J, Zendejas B, Szostek J H, Erwin P J, et al. Comparative effectiveness of technology-enhanced simulation versus other instructional methods: a systematic review and meta-analysis. Simul Healthc 2012; 7(5): 308-20. doi: 10.1097/SIH.0b013e3182614f95. DOI: https://doi.org/10.1097/SIH.0b013e3182614f95
Gustafsson A, Viberg B, Paltved C, Palm H, Konge L, Nayahangan L J. Identifying technical procedures in orthopaedic surgery and traumatology that should be integrated in a simulation-based curriculum: a national general needs assessment in Denmark. J Bone Joint Surg Am 2019; 101(20): e108. doi: 10.2106/JBJS.18.01122. DOI: https://doi.org/10.2106/JBJS.18.01122
Blyth P, Stott N S, Anderson I A. Virtual reality assessment of technical skill using the Bonedoc DHS simulator. Injury 2008; 39(10): 1127–33. doi: 10.1002/bjs.10313. DOI: https://doi.org/10.1016/j.injury.2008.02.006
Gustafsson A, Pedersen P, Rømer T B, Viber B, Palm H, Konge L. Hip-fracture osteosynthesis training: exploring learning curves and setting proficiency standards. Acta Orthop 2019; 90(4): 348-53. doi: 10.1080/17453674.2019.1607111. DOI: https://doi.org/10.1080/17453674.2019.1607111
Rambani R, Viant W, Ward J Mohsen A. Computer-assisted orthopedic training system for fracture fixation. J Surg Educ 2013; 70(3): 304-8. doi: 10.1016/j.jsurg.2012.11.009. DOI: https://doi.org/10.1016/j.jsurg.2012.11.009
Rahnish R K, Srivastava A, Kumar P, Yadav S K, Sharma S, Hag R U, et al. Comparison of outcomes of long versus short cephalomedullary nails for the fixation of intertrochanteric femur fractures: a systematic review and meta-analysis of 14,547 patients. Indian J Orthop. 2023;57(8): 1165-87. doi: 10.1007/s43465-023-00915-5. DOI: https://doi.org/10.1007/s43465-023-00915-5
Cook D A, Brydges R, Zendejas B, Hamstra S J, Hatala R. Mastery learning for health professionals using technology-enhanced simulation: a systematic review and meta-analysis. Acad Med 2013; 88(8): 1178-86. doi: 10.1097/ACM.0b013e31829a365d. DOI: https://doi.org/10.1097/ACM.0b013e31829a365d
Cook D A, Hatala R. Validation of educational assessments: a primer for simulation and beyond. Adv Simul(Lond) 2016; 1: 31. doi: 10.1186/s41077-016-0033-y. DOI: https://doi.org/10.1186/s41077-016-0033-y
Goldberg M G, Garbens A, Szasz P, Hauer T, Grantcharov T P. Systematic review to establish absolute standards for technical performance in surgery. Br J Surg 2017; 104: 13-21. doi: 10.1002/bjs.10313. DOI: https://doi.org/10.1002/bjs.10313
Messick S. Validity. In: Linn RL, editor. Educational measurement. New York: American Macmillan Publishing; 1989.
Schouten H J A. Sample size formula with a continuous outcome for unequal group sizes and unequal variances. Stat Med 1999; 18(1): 87-91. doi: 10.1002/(sici)1097-0258(19990115)18:1<87::aid-sim958>3.0.co;2-k. DOI: https://doi.org/10.1002/(SICI)1097-0258(19990115)18:1<87::AID-SIM958>3.0.CO;2-K
Yudkowsky R, Downing S M, Tekian A. Standard setting. In: Assessment in health professions education. New York: Routledge; 2009. Available from: https://doi.org/10.4324/9780203880135. DOI: https://doi.org/10.4324/9780203880135
Andersen S A, Konge L, Mikkelsen P T, Cayé-Thomasen P, Sørensen M S. Mapping the plateau of novices in virtual reality simulation training of mastoidectomy. Laryngoscope 2017; 127: 907-14. doi: 10.1002/lary.26000. DOI: https://doi.org/10.1002/lary.26000
Dyre L, Nørgaard L N, Tabor A, Madsen M E, Sørensen J L, Ringsted C, et al. Collecting validity evidence for the assessment of mastery learning in simulation-based ultrasound training. Ultraschall in Med 2015; 36: 1-7. doi: 10.1002/jum.14292. DOI: https://doi.org/10.1002/jum.14292
Madsen M E, Konge L, Nørgaard L N, Tabor A, Ringsted C, Klemmensen Å K, et al. Assessment of performance measures and learning curves for use of a virtual-reality ultrasound simulator in transvaginal ultrasound examination. Ultrasound Obstet Gynecol 2014; 44: 693-9. doi: 10.1002/uog.13400. DOI: https://doi.org/10.1002/uog.13400
Cook D A, Artino A R. Motivation to learn: an overview of contemporary theories. Med Educ 2016; 50: 997-1014. doi: 10.1111/medu.13074. DOI: https://doi.org/10.1111/medu.13074
Schmidt R A, Bjork R A. New conceptualizations of practice: common principles in three paradigms suggest new concepts for training. Psychol Sci 1992; 3: 207-17. doi: 10.1177/1745691617690642. DOI: https://doi.org/10.1111/j.1467-9280.1992.tb00029.x
Dyre L, Tabor A, Ringsted C, Tolsgaard M. Imperfect practice makes perfect: error management training improves transfer of learning. Med Educ 2017; 51: 196-206. doi: 10.1111/medu.13208. DOI: https://doi.org/10.1111/medu.13208
Rölfing J, Sloth S B, Falstie-Jensen T, Mygind-Klavsen B, Elsøe R, Jensen R D. Opportunity to perform: simulation-based surgical training should be provided on demand. J Med Educ Curric Dev 2023: 10: 23821205231219429. doi: 1177/23821205231219429. DOI: https://doi.org/10.1177/23821205231219429
Konge L, Arendrup H, von Buchwald C, Ringsted C. Using performance in multiple simulated scenarios to assess bronchoscopy skills. Respiration 2011; 81(6): 483-90. doi: 10.1159/000324452. DOI: https://doi.org/10.1159/000324452
Østergaard M L, Nielsen K R, Albrecht-Beste E, Konge L, Nielsen M B. Development of a reliable simulation-based test for diagnostic abdominal ultrasound with a pass/fail standard usable for mastery learning. Eur Radiol 2018; 28(1): 51-7. doi: 10.1007/s00330-017-4913-x. DOI: https://doi.org/10.1007/s00330-017-4913-x
Cold K M, Konge L, Clementsen P F, Nayahangan L J. Simulation-based mastery learning of flexible bronchoscopy: deciding factors for completion. Respiration 2019; 97(2): 160-7. doi: 10.1159/000493431. DOI: https://doi.org/10.1159/000493431
Rölfing J D, Jensen R D, Paltved C. HipSim: hip fracture surgery simulation utilizing the Learning Curve–Cumulative Summation test (LC-CUSUM). Acta Orthop 2020; 91(6): 669-74. doi: 10.1080/17453674.2020.1777511. DOI: https://doi.org/10.1080/17453674.2020.1777511
Nayahangan L J, Thinggaard E, Khan F, Gustafsson A, Mørcke A M, Dubrowski A, et al. A view from the top: a qualitative exploration of top-level health care leaders’ perceptions on the implementation of simulation-based education in postgraduate medical education. Med Educ 2024; 58(4): 415-29. doi: 10.1111/medu.15248. DOI: https://doi.org/10.1111/medu.15248
Additional Files
Published
How to Cite
License
Copyright (c) 2024 Amandus Gustafsson , Jan D Rölfing, Henrik Palm, Bjarke Viberg, Søren Grimstrup, Lars Konge
This work is licensed under a Creative Commons Attribution 4.0 International License.
