How reliable are reliability studies of fracture classifications?
A systematic review of their methodologies

References

• Altman D G. Inter-rater agreement. In: Practical statistics for medical research. Chapman & Hall / CRC London 1990: 403–9.
   Google Scholar
• Andersen D J, Blair W F, Steyers C M, Jr., Adams B D, el Khouri G Y, Brandser E A. Classification of distal radius fractures: an analysis of interobserver reliability and intraobserver reproducibility. J Hand Surg (Am) 1996; 21(4): 574–82.
   Google Scholar
• Andersen E, Jorgensen L G, Hededam L T. Evans' clas-sification of trochanteric fractures: an assessment of the interobserver and intraobserver reliability. Injury 1990; 21(6): 377–8.
   PubMed Web of Science ®Google Scholar
• Andersen G R, Rasmussen J B, Dahl B, Solgaard S. Older 's classification of Cones' fractures. Good intraobserver and interobserver reproducibility in 185 cases. Acta Orthop Scand 1991; 62 (5): 463–4.
   Google Scholar
• Barton K L, Kaminsky C K, Green D W, Shean C J, Kautz S M, Skaggs DL. Reliability of a modified Gartland clas-sification of supracondylar humerus fractures. J Pediatr Orthop 2001; 21(1): 27–30.
   PubMed Web of Science ®Google Scholar
• Bernstein J, Adler L M, Blank J E, Dalsey R M, Williams G R, Iannotti J P. Evaluation of the Neer system of classifi-cation of proximal humeral fractures with computerized tomographic scans and plain radiographs. J Bone Joint Surg (Am) 1996; 78 (9): 1371–5.
   PubMed Web of Science ®Google Scholar
• Bernstein J, Monaghan B A, Silber J S, DeLong W G. Tax-onomy and treatment-a classification of fracture classifi-cations. J Bone Joint Surg (Br) 1997; 79 (5): 706–7.
   PubMedGoogle Scholar
• Bjorgul K, Reikeraos 0. Low interobserver reliability of radiographic signs predicting healing disturbance in displaced intracapsular fracture of the femoral neck. Acta Orthop Scand 2002; 73 (3): 307–10.
   Google Scholar
• Bland J M, Altman D G. Statistics notes: Validating scales and indexes. BMJ 2002; 324 (7337): 606–7.
   PubMed Web of Science ®Google Scholar
• Blauth M, Bastian L, Knop C, Lange U, Tusch G. [Inter-observer reliability in the classification of thoraco-lumbar spinal injuries]. Orthopäde 1999; 28 (8): 662–81.
   Google Scholar
• Blundell C M, Parker M J, Pryor G A, Hopkinson-Wool-ley J, Bhonsle S S. Assessment of the AO classification of intracapsular fractures of the proximal femur. J Bone Joint Surg (Br) 1998; 80 (4): 679–83.
   Google Scholar
• Brady 0 H, Garbuz D S, Masri B A, Duncan C P. The reliability and validity of the Vancouver classification of femoral fractures after hip replacement. J Arthroplasty 2000; 15 (1): 59–62.
   Google Scholar
• Brage ME, Rockett M, Vraney R, Anderson R, Toledano A. Anlde fracture classification: a comparison of reliability of three X-ray views versus two. Foot Ankle hit 1998; 19 (8): 555–62.
   Google Scholar
• Brien H, Noftall F, MacMaster S, Cummings T, Landells C, Rockwood P. Neer 's classification system: a critical appraisal. J Trauma 1995; 38 (2): 257–60.
   PubMedGoogle Scholar
• Brorson S, Bagger J, Sylvest A, Hobjartsson A. Improved interobserver variation after training of doctors in the Neer system. A randomised trial. J Bone Joint Surg (Br) 2002; 84 (7): 950–4.
   PubMed Web of Science ®Google Scholar
• Browner B D, Jupiter J B, Levine AM, Trafton PG. Skeletal trauma - fractures, dislocations, ligamentous injuries. W.B. Saunders Company, A Division of Harcourt Brace & Company Philadelphia, London, Toronto, Montreal, Sydney, Tokyo 1998.
   Google Scholar
• Burstein All. Fracture classification systems: do they work and are they useful? J Bone Joint Surg (Am) 1993; 75 (12): 1743–4.
   PubMed Web of Science ®Google Scholar
• Chan P S, IClimIdewicz J J, Luchetti W T, Esterhai J L, Kneeland J B, Dalinka M K, Heppenstall R B. Impact of CT scan on treatment plan and fracture classification of tibial plateau fractures. J Orthop Trauma 1997; 11(7): 484–9.
   PubMed Web of Science ®Google Scholar
• Colton C L. Telling the bones. J Bone Joint Surg (Br) 1991; 73 (3): 362–4.
   PubMedGoogle Scholar
• Cook R J. Kappa and its dependence on marginal rates. hi: Encyclopedia of biostatistics (Eds. Armitage P and Colton T). Wiley New York 1998: 2166–8.
   Google Scholar
• Craig W L, BI, Dirschl D R. Effects of binary decision making on the classification of fractures of the ankle. J Orthop Trauma 1998; 12 (4): 280–3.
   Google Scholar
• Dirschl D R, Adams G L. A critical assessment of factors influencing reliability in the classification of fractures, using fractures of the tibial plafond as a model. J Orthop Trauma 1997; 11(7): 471–6.
   PubMed Web of Science ®Google Scholar
• Fleiss J L. Statistical methods for rates and proportions, 2nd ed. hi: John Wiley & Sons New York 1981: 217–8.
   Google Scholar
• Flinkkild T, Nikkola-Sihto A, Kaarela 0, Paakko E, Raati-kainen T. Poor interobserver reliability of AO classifica-tion of fractures of the distal radius. Additional computed tomography is of minor value. J Bone Joint Surg (Br) 1998; 80 (4): 670–2.
   Google Scholar
• Frandsen P A, Andersen E, Madsen F, Skjodt T. Garden's classification of femoral neck fractures. An assessment of inter-observer variation. J Bone Joint Surg (Br) 1988; 70 (4): 588–90.
   PubMed Web of Science ®Google Scholar
• Gehrchen P M, Nielsen J 0, Olesen B. Poor reproducibil-ity of Evans' classification of the trochanteric fracture. Assessment of 4 observers in 52 cases. Acta Orthop Scand 1993; 64(1): 71–2.
   Google Scholar
• Gehrchen PM, Nielsen J 0, Olesen B, Andresen B K. Sein-sheimer 's classification of subtrochanteric fractures. Poor reproducibility of 4 observers' evaluation of 50 cases. Acta Orthop Scand 1997; 68 (6): 524–6.
   Google Scholar
• Gjorup T. The kappa coefficient and the prevalence of a diagnosis. Methods Inf Med 1988; 27 (4): 184–6.
   PubMed Web of Science ®Google Scholar
• Illarramendi A, Gonzalez D, V, Segal E, De Carli P, Maignon G, Gallucci G. Evaluation of simplified Frykman and AO classifications of fractures of the distal radius. Assessment of interobserver and intraobserver agreement. hit Orthop 1998; 22(2): 111–5.
   Google Scholar
• Johnstone D J, Radford W J, Parnell E J. Interobserver variation using the AO/ASIF classification of long bone fractures. Injury 1993; 24 (3): 163–5.
   PubMed Web of Science ®Google Scholar
• Kreder H J, Hanel D P, McKee M, Jupiter J, McGillivary G, Swiontkowski M E Consistency of AO fracture classifica-tion for the distal radius. J Bone Joint Surg (Br) 1996; 78 (5): 726–31.
   PubMedGoogle Scholar
• Kristiansen B, Andersen U L, Olsen CA, Varmarken J E. The Neer classification of fractures of the proximal humerus. An assessment of interobserver variation. Skeletal Radiol 1988; 17 (6): 420–2.
   PubMed Web of Science ®Google Scholar
• Landis J R, Koch G G. The measurement of observer agreement for categorical data. Biometrics 1977; 33 (1): 159–74.
   PubMed Web of Science ®Google Scholar
• Lichtenhahn P, Fernandez DL, Schatzker J. [Analysis of the "user friendliness" of the AO classification of fractures]. Hely Chir Acta 1992; 58(6): 919–24.
   PubMed Web of Science ®Google Scholar
• Lindsjo U. Classification of anlde fractures: the Lauge-Hansen or AO system? Clin Orthop 1985; (199): 12–6.
   Google Scholar
• Martin J S, Marsh J L. Current classification of fractures. Rationale and utility. Radiol Chin North Am 1997; 35 (3): 491–506.
   PubMed Web of Science ®Google Scholar
• Martin J S, Marsh J L, Bonar S K, DeCoster TA, Found E M, Brandser E A. Assessment of the AO/ASIF fracture classification for the distal tibia. J Orthop Trauma 1997; 11(7): 477–83.
   PubMed Web of Science ®Google Scholar
• McAdams T R, Blevins F T, Martin T P, DeCoster TA. The role of plain films and computed tomography in the evalu-ation of scapular neck fractures. J Orthop Trauma 2002; 16(1): 7–11.
   PubMed Web of Science ®Google Scholar
• Morgan S J, Groshen S L, Itamura J M, Shankwller J, Brien W W, Kuschner S H. Reliability evaluation of classifying radial head fractures by the system of Mason. Bull Hosp Jt Dis 1997; 56 (2): 95–8.
   PubMedGoogle Scholar
• Nielsen J 0, Dons-Jensen H, Sorensen H T. Lauge-Hansen classification of malleolar fractures. An assessment of the reproducibility in 118 cases. Acta Orthop Scand 1990; 61 (5): 385–7.
   Google Scholar
• Oner F C, Ramos L M, Simmermacher R K, Kingma P T, Diekerhof C H, Dhert W J, Verbout A J. Classification of thoracic and lumbar spine fractures: problems of repro-ducibility. A study of 53 patients using CT and MRI. Eur Spine J 2002; 11(3): 235–45.
   Google Scholar
• Oskam J, Kingma J, Klasen H J. Interrater reliability for the basic categories of the AO/ASIF's system as a frame of reference for classifying distal radial fractures. Percept Mot Skills 2001; 92(2): 589–94.
   Google Scholar
• Pervez H, Parker M, Pryor G, Lutchman L, Chirodian N. Classification of trochanteric fracture of the proximal femur: a study of the reliability of current systems. Injury 2002; 33 (8): 713–5.
   Google Scholar
• Pewsner D, Battaglia M, Bucher H, Minder C, Grossen-bacher F, Egger M. The Bayes Library of Diagnostic Studies and Reviews. Basel Institute for Clinical Epi-demiology, Institute for Social and Preventive Medicine University of Berne Basel, Berne 2002.
   Google Scholar
• Rasmussen S, Madsen PV, Bennicke K. Observer variation in the Lauge-Hansen classification of anIde fractures. Precision improved by instruction. Acta Orthop Scand 1993; 64 (6): 693–4.
   Google Scholar
• Rockwood C A, Green D P, Bucholz R W, Heckman J D. Rockwood and Green's Fractures in Adults. Lippincott-Raven Philadelphia, New York 1996.
   Google Scholar
• Schipper I B, Steyerberg E W, Castelein R M, van Vugt A B. Reliability of the AO/ASIF classification for pertro-chanteric femoral fractures. Acta Orthop Scand 2001; 72(1): 36–41.
   PubMedGoogle Scholar
• Shrout P E. Measurement reliability and agreement in psy-chiatry. Stat Methods Med Res 1998; 7 (3): 301–17.
   PubMedGoogle Scholar
• Sidor M L, Zuckerman J D, Lyon T, Koval K, Cuomo F, Schoenberg N. The Neer classification system for proxi-mal humeral fractures. An assessment of interobserver reliability and intraobserver reproducibility. J Bone Joint Surg (Am) 1993; 75 (12): 1745–50.
   Google Scholar
• Siebenrock K A, Gerber C. The reproducibility of classifica-tion of fractures of the proximal end of the humerus. J Bone Joint Surg (Am) 1993; 75 (12): 1751–5.
   PubMed Web of Science ®Google Scholar
• Silber J S, Flynn J M, Katz MA, Ganley T J, Koffler K M, Drummond D S. Role of computed tomography in the classification and management of pediatric pelvic frac-tures. J Pediatr Orthop 2001; 21(2): 148–51.
   PubMed Web of Science ®Google Scholar
• SY:idén G 0, Movin T, Guntner P, Aspelin P, Ahrengart L, Ersmark H, Sperber A. Poor reproducibility of classifica-tion of proximal humeral fractures. Additional CT of minor value. Acta Orthop Scand 1997; 68 (3): 239–42.
   Google Scholar
• SY:idén G 0, Movin T, Aspelin P, Guntner P, Shalabi A. 3D-radiographic analysis does not improve the Neer and AO classifications of proximal humeral fractures. Acta Orthop Scand 1999; 70 (4): 325–8.
   Google Scholar
• Svanholm H, Starklint H, Gundersen H J, Fabricius J, Bar-lebo H, Olsen S. Reproducibility of histomorphologic diagnoses with special reference to the kappa statistic. APMIS 1989; 97 (8): 689–98.
   Google Scholar
• Swiontkowski M F, Sands AK, Agel J, Diab M, Schwap-pach J R, Kreder H J. Interobserver variation in the AO/ OTA fracture classification system for pilon fractures: is there a problem? J Orthop Trauma 1997; 11(7): 467–70.
   Google Scholar
• Thomsen N 0, Overgaard S, Olsen L H, Hansen H, Nielsen S T. Observer variation in the radiographic classification of anIde fractures. J Bone Joint Surg (Br) 1991; 73 (4): 676–8.
   Google Scholar
• Thomsen N 0, Jensen C M, Skovgaard N, Pedersen M S, Pallesen P, Soe-Nielsen N H, Rosenldint A. Observer variation in the radiographic classification of fractures of the neck of the femur using Garden's system. hit Orthop 1996; 20 (5): 326–9.
   Google Scholar
• Thomsen N 0, Olsen L H, Nielsen S T. Kappa statistic in the assessment of observer variation: the significance of multiple observers classifying anlde fractures. J Orthop Sci 2002; 7 (2): 163–6.
   Google Scholar
• Uebersax J. Kappa Coefficients. http://ourworld.compuserve. com/homepages/jsuebersax/kappa.htm 2002a.
   Google Scholar
• Uebersax J. Statistical Methods for Rater Agreement. http: //ourworld.compuserve.com/homepages/jsuebersax/ agree.htm 2002b.
   Google Scholar
• Visutipol B, Chobtangsin P, Ketmalasiri B, Pattarabanjird N, Varodompun N. Evaluation of Letournel and Judet clas-sification of acetabular fracture with plain radiographs and three-dimensional computerized tomographic scan. J Orthop Surg 2000; 8 (1): 33–7.
   Google Scholar
• Wainwright AM, Williams J R, Can A J. Interobserver and intraobserver variation in classification systems for frac-tures of the distal humerus. J Bone Joint Surg (Br) 2000; 82 (5): 636–42.
   PubMedGoogle Scholar
• Yacoubian S V, Nevins R T, Sallis J G, Potter HG, Lorich D G. Impact of MRI on treatment plan and fracture classifi-cation of tibial plateau fractures. J Orthop Trauma 2002; 16 (9): 632–7.
   PubMed Web of Science ®Google Scholar