Clinical prediction models for patients undergoing total hip arthroplasty: an external validation based on a systematic review and the Dutch Arthroplasty Register
DOI:
https://doi.org/10.2340/17453674.2024.42449Keywords:
Arthroplasty, External validation, Hip, Prediction modelAbstract
Background and purpose: External validation is a crucial step after prediction model development. Despite increasing interest in prediction models, external validation is frequently overlooked. We aimed to evaluate whether joint registries can be utilized for external validation of prediction models, and whether published prediction models are valid for the Dutch population with a total hip arthroplasty.
Methods: We identified prediction models developed in patients undergoing arthroplasty through a systematic literature search. Model variables were evaluated for availability in the Dutch Arthroplasty Registry (LROI). We assessed the model performance in terms of calibration and discrimination (area under the curve [AUC]). Furthermore, the models were updated and evaluated through intercept recalibration and logistic recalibration.
Results: After assessing 54 papers, 19 were excluded for not describing a prediction model (n = 16) or focusing on non-TJA populations (n = 3), leaving 35 papers describing 44 prediction models. 90% (40/44) of the prediction models used outcomes or predictors missing in the LROI, such as diabetes, opioid use, and depression. 4 models could be externally validated on LROI data. The models’ discrimination ranged between poor and acceptable and was similar to that in the development cohort. The calibration of the models was insufficient. The model performance improved slightly after updating.
Conclusion: External validation of the 4 models resulted in suboptimal predictive performance in the Dutch population, highlighting the importance of external validation studies.
Downloads
References
Garland A, Bulow E, Lenguerrand E, Blom A, Wilkinson M, Sayers A, et al. Prediction of 90-day mortality after total hip arthroplasty. Bone Joint J 2021; 103-B(3): 469-78. doi: 10.1302/0301-620X.103B3.BJJ-2020-1249.R1. DOI: https://doi.org/10.1302/0301-620X.103B3.BJJ-2020-1249.R1
Paxton E W, Inacio M C, Khatod M, Yue E, Funahashi T, Barber T. Risk calculators predict failures of knee and hip arthroplasties: findings from a large health maintenance organization. Clin Orthop Relat Res 2015; 473(12): 3965-73. doi: 10.1007/s11999-015-4506-4. DOI: https://doi.org/10.1007/s11999-015-4506-4
Venalainen M S, Panula V J, Klen R, Haapakoski J J, Eskelinen A P, Manninen M J, et al. Preoperative risk prediction models for short-term revision and death after total hip arthroplasty: data from the Finnish Arthroplasty Register. JBJS Open Access 2021; 6(1). doi: 10.2106/JBJS.OA.20.00091. DOI: https://doi.org/10.2106/JBJS.OA.20.00091
Harris A H S, Kuo A C, Weng Y, Trickey A W, Bowe T, Giori N J. Can machine learning methods produce accurate and easy-to-use prediction models of 30-day complications and mortality after knee or hip arthroplasty? Clin Orthop Relat Res 2019; 477(2): 452-60. doi: 10.1097/CORR.0000000000000601. DOI: https://doi.org/10.1097/CORR.0000000000000601
Everhart J S, Andridge R R, Scharschmidt T J, Mayerson J L, Glassman A H, Lemeshow S. Development and validation of a preoperative surgical site infection risk score for primary or revision knee and hip arthroplasty. J Bone Joint Surg Am 2016; 98(18): 1522-32. doi: 10.2106/JBJS.15.00988. DOI: https://doi.org/10.2106/JBJS.15.00988
Tan T L, Maltenfort M G, Chen A F, Shahi A, Higuera C A, Siqueira M, et al. Development and evaluation of a preoperative risk calculator for periprosthetic joint infection following total joint arthroplasty. J Bone Joint Surg Am 2018; 100(9): 777-85. doi: 10.2106/JBJS.16.01435. DOI: https://doi.org/10.2106/JBJS.16.01435
Cochrane J A, Flynn T, Wills A, Walker F R, Nilsson M, Johnson S J. Clinical decision support tools for predicting outcomes in patients undergoing total knee arthroplasty: a systematic review. J Arthroplasty 2021; 36(5): 1832-45.e1. doi: 10.1016/j.arth.2020.10.053. DOI: https://doi.org/10.1016/j.arth.2020.10.053
Bouwmeester W, Zuithoff N P, Mallett S, Geerlings M I, Vergouwe Y, Steyerberg E W, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med 2012; 9(5): 1-12. doi: 10.1371/journal.pmed.1001221. DOI: https://doi.org/10.1371/journal.pmed.1001221
Ramspek C L, Jager K J, Dekker F W, Zoccali C, van Diepen M. External validation of prognostic models: what, why, how, when and where? Clin Kidney J 2021; 14(1): 49-58. doi: 10.1093/ckj/sfaa188. DOI: https://doi.org/10.1093/ckj/sfaa188
Riley R D, Ensor J, Snell K I, Debray T P, Altman D G, Moons K G, et al. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 2016; 353:i3140. doi: 10.1136/bmj.i3140. DOI: https://doi.org/10.1136/bmj.i3140
Collins G S, Reitsma J B, Altman D G, Moons K G. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 2015; 13:1. doi: 10.1186/s12916-014-0241-z. DOI: https://doi.org/10.1186/s12916-014-0241-z
LROI. LROI rapportage 2022. Available at: https://www.lroi.nl/media/3j2o5wjg/pdf-lroi-annual-report-2022.pdf
Riley R D, Debray T P A, Collins G S, Archer L, Ensor J, van Smeden M, et al. Minimum sample size for external validation of a clinical prediction model with a binary outcome. Stat Med 2021; 40(19): 4230-51. doi: 10.1002/sim.9025. DOI: https://doi.org/10.1002/sim.9025
Vergouwe Y, Steyerberg E W, Eijkemans M J, Habbema J D. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 2005; 58(5): 475-83. doi: 10.1016/j.jclinepi.2004.06.017. DOI: https://doi.org/10.1016/j.jclinepi.2004.06.017
Bonsel J M, Reijman M, Verhaar J A N, van Steenbergen L N, Janssen M F, Bonsel G J. Socioeconomic inequalities in patient-reported outcome measures of Dutch primary hip and knee arthroplasty patients for osteoarthritis. Osteoarthritis Cartilage 2024; 32(2): 200-9. doi: 10.1016/j.joca.2023.07.004. DOI: https://doi.org/10.1016/j.joca.2023.07.004
Hosmer D, Lemeshow S. Applied logistic regression: Chichester: Wiley; 2000. DOI: https://doi.org/10.1002/0471722146
Van Calster B, McLernon D J, van Smeden M, Wynants L, Steyerberg E W, Topic Group “Evaluating diagnostic tests and prediction models”, et al. Calibration: the Achilles heel of predictive analytics. BMC Med 2019; 17(1): 230. doi: 10.1186/s12916-019-1466-7. DOI: https://doi.org/10.1186/s12916-019-1466-7
Van Calster B, Van Hoorde K, Vergouwe Y, Bobdiwala S, Condous G, Kirk E, et al. Validation and updating of risk models based on multinomial logistic regression. Diagn Progn Res 2017; 1:2. doi: 10.1186/s41512-016-0002-x. DOI: https://doi.org/10.1186/s41512-016-0002-x
R Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2021.
Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina M J, Steyerberg E W. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 2016; 74: 167-76. doi: 10.1016/j.jclinepi.2015.12.005. DOI: https://doi.org/10.1016/j.jclinepi.2015.12.005
Harrell Jr F. rms: Regression modeling strategies 2022. Available from: https://CRAN.R-project.org/package=rms.
Groot O Q, Bindels B J J, Ogink P T, Kapoor N D, Twining P K, Collins A K, et al. Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review. Acta Orthop 2021; 92(4): 385-93. doi: 10.1080/17453674.2021.1910448. DOI: https://doi.org/10.1080/17453674.2021.1910448
Slieker R C, van der Heijden A, Siddiqui M K, Langendoen-Gort M, Nijpels G, Herings R, et al. Performance of prediction models for nephropathy in people with type 2 diabetes: systematic review and external validation study. BMJ 2021; 374:n2134. doi: 10.1136/bmj.n2134. DOI: https://doi.org/10.1136/bmj.n2134
Hueting T A, van Maaren M C, Hendriks M P, Koffijberg H, Siesling S. External validation of 87 clinical prediction models supporting clinical decisions for breast cancer patients. Breast 2023; 69:382-91. doi: 10.1016/j.breast.2023.04.003. DOI: https://doi.org/10.1016/j.breast.2023.04.003
John L H, Kors J A, Fridgeirsson E A, Reps J M, Rijnbeek P R. External validation of existing dementia prediction models on observational health data. BMC Med Res Methodol 2022; 22(1): 311. doi: 10.1186/s12874-022-01793-5. DOI: https://doi.org/10.1186/s12874-022-01793-5
Vickers A J, van Calster B, Steyerberg E W. A simple, step-by-step guide to interpreting decision curve analysis. Diagn Progn Res 2019; 3:18. doi: 10.1186/s41512-019-0064-7. DOI: https://doi.org/10.1186/s41512-019-0064-7
Moons K G, Kengne A P, Grobbee D E, Royston P, Vergouwe Y, Altman D G, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012; 98(9): 691-8. doi: 10.1136/heartjnl-2011-301247. DOI: https://doi.org/10.1136/heartjnl-2011-301247
Wallisch C, Heinze G, Rinner C, Mundigler G, Winkelmayer W C, Dunkler D. Re-estimation improved the performance of two Framingham cardiovascular risk equations and the Pooled Cohort equations: a nationwide registry analysis. Sci Rep 2020; 10(1): 8140. doi: 10.1038/s41598-020-64629-6. DOI: https://doi.org/10.1038/s41598-020-64629-6
Silman A J, Combescure C, Ferguson R J, Graves S E, Paxton E W, Frampton C, et al. International variation in distribution of ASA class in patients undergoing total hip arthroplasty and its influence on mortality: data from an international consortium of arthroplasty registries. Acta Orthop 2021; 92(3): 304-10. doi: 10.1080/17453674.2021.1892267. DOI: https://doi.org/10.1080/17453674.2021.1892267
Luijken K, Song J, Groenwold R H H. Quantitative prediction error analysis to investigate predictive performance under predictor measurement heterogeneity at model implementation. Diagn Progn Res 2022; 6(1): 7. doi: 10.1186/s41512-022-00121-1. DOI: https://doi.org/10.1186/s41512-022-00121-1
Rolfson O. Editorial comment: 7th International Congress of Arthroplasty Registries. Clin Orthop Relat Res 2019; 477(6): 1299-300. doi: 10.1097/CORR.0000000000000796. DOI: https://doi.org/10.1097/CORR.0000000000000796
Lubbeke A, Silman A J, Barea C, Prieto-Alhambra D, Carr A J. Mapping existing hip and knee replacement registries in Europe. Health Policy 2018; 122(5): 548-57. doi: 10.1016/j.healthpol.2018.03.010. DOI: https://doi.org/10.1016/j.healthpol.2018.03.010
Steyerberg E W, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J 2014; 35(29): 1925-31. doi: 10.1093/eurheartj/ehu207. DOI: https://doi.org/10.1093/eurheartj/ehu207
Additional Files
Published
How to Cite
License
Copyright (c) 2024 Maartje Belt, Katrijn Smulders, B Willem Schreurs, Gerjon Hannink
This work is licensed under a Creative Commons Attribution 4.0 International License.
