A simplified, 2-question grading system for evaluating abstracts in orthopedic scientific meetings: a serial randomization study
DOI:
https://doi.org/10.2340/17453674.2024.40504Keywords:
abstract, conference, intraclass correlation coefficients, scoring systemAbstract
Background and purpose: Efficient abstract scoring for congress presentation is important. Given the emergence of new study methodologies, a scoring system that accommodates all study designs is warranted. We aimed to assess the equivalence of a simplified, 2-question abstract grading system with a more complex currently used system in assessing abstracts submitted for orthopedic scientific meetings in a serial randomized study.
Methods: Dutch Orthopedic Association Scientific Committee (DOASC) members were randomized to grade abstracts using either the current grading system, which includes up to 7 scoring categories, or the new grading system, which consists of only 2 questions. Pearson correlation coefficient and mean abstract score with 95% confidence intervals (CI) were calculated.
Results: Analysis included the scoring of 195 abstracts by 12–14 DOASC members. The average score for an abstract using the current system was 60 points (CI 58–62), compared with 63 points (CI 62–64) using the new system. By using the new system, abstracts were scored higher by 3.3 points (CI 1.7–5.0). Pearson correlation was poor with coefficient 0.38 (P < 0.001).
Conclusion: The simplified abstract grading system exhibited a poor correlation with the current scoring system, while the new system offers a more inclusive evaluation of varying study designs and is preferred by almost all DOASC members.
Downloads
References
Kuczmarski T M, Raja A S, Pallin D J. How do medical societies select science for conference presentation? How should they? West J Emerg Med 2015; 16: 543-50. doi: 10.5811/westjem.2015.5.25518. DOI: https://doi.org/10.5811/westjem.2015.5.25518
Rowe B H, Strome T L, Spooner C, Blitz S, Grafstein E, Worster A. Reviewer agreement trends from four years of electronic submissions of conference abstract. BMC Med Res Methodol 2006; 6: 14. doi: 10.1186/1471-2288-6-14. DOI: https://doi.org/10.1186/1471-2288-6-14
Montgomery A A, Graham A, Evans P H, Fahey T. Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference. BMC Health Serv Res 2002; 2: 8. doi: 10.1186/1472-6963-2-8. DOI: https://doi.org/10.1186/1472-6963-2-8
Timmer A, Sutherland L R, Hilsden R J. Development and evaluation of a quality score for abstracts. BMC Med Res Methodol 2003; 3: 2. doi: 10.1186/1471-2288-3-2. DOI: https://doi.org/10.1186/1471-2288-3-2
Rahbek O, Jensen S L, Lind M, Penny J O, Kallemose T, Jakobsen T, et al. Inferior reliability of VAS scoring compared with International Society of the Knee reporting system for abstract assessment. Dan Med J 2017; 64(4): A5346. PMID: 28385168.
Poolman R W, Keijser L C, de Waal Malefijt M C, Blankevoort L, Farrokhyar F, Bhandari M, et al. Reviewer agreement in scoring 419 abstracts for scientific orthopedics meetings. Acta Orthop 2007; 78: 278-84. doi: 10.1080/17453670710013807. DOI: https://doi.org/10.1080/17453670710013807
Dwan K, Li T, Altman D G, Elbourne D. CONSORT 2010 statement: extension to randomised crossover trials. BMJ 2019; 366: l4378. doi: 10.1136/bmj.l4378. DOI: https://doi.org/10.1136/bmj.l4378
Shrout P E, Fleiss J L. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979; 86: 420-8. doi: 10.1037//0033-2909.86.2.420. DOI: https://doi.org/10.1037//0033-2909.86.2.420
Koo T K, Li M Y. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016; 15: 155-63. doi: 10.1016/j.jcm.2016.02.012. DOI: https://doi.org/10.1016/j.jcm.2016.02.012
Mitchell N S, Stolzmann K, Benning L V, Wormwood J B, Linsky A M. Effect of a scoring rubric on the review of scientific meeting abstracts. J Gen Intern Med 2020; 36: 2483-5. doi: 10.1007/s11606-020-05960-6. DOI: https://doi.org/10.1007/s11606-020-05960-6
Bydder S, Marion K, Taylor M, Semmens J. Assessment of abstracts submitted to the annual scientific meeting of the Royal Australian and New Zealand College of Radiologists. Australas Radiol 2006; 50: 355-9. doi: 10.1111/j.1440-1673.2006.01599.x. DOI: https://doi.org/10.1111/j.1440-1673.2006.01599.x
Additional Files
Published
How to Cite
License
Copyright (c) 2024 Walter van der Weegen, Jeroen C van Egmond, Ruth E Geuze, Taco Gosens, Barbara Snoeker, Rudolf W Poolman
This work is licensed under a Creative Commons Attribution 4.0 International License.
