Inter-Rater Agreement and Reliability of the COSMIN (COnsensus-Based Standards for the Selection of Health Status Measurement Instruments) Checklist

Details

• Personal Author:
  Alonso J ; Bouter LM ; de Vet HCW ; Gibbons E ; Knol DL ; Mokkink LB ; Patrick DL ; Stratford PW ; Terwee CB
• Description:
  Background: The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114). Methods: 75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item. Results: 88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions. Conclusions: Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved. [Description provided by NIOSH]
• Subjects:
  Occupational Health Safety Surveys And Questionnaires
• Keywords:
  Health Care Health Surveys Standards Statistical Analysis Statistical Quality Control
• ISSN:
  1471-2288
• Document Type:
  Text
• Funding:
  Grant
• Genre:
  Journal Article
• Place as Subject:
  OSHA Region 10 ; Washington
• CIO:
  National Institute for Occupational Safety and Health (NIOSH)
• Topic:
  Workplace Safety & Health
• Location:
  North America
• Pages in Document:
  82
• Volume:
  10
• NIOSHTIC Number:
  nn:20056977
• Citation:
  BMC Med Res Methodol 2010 Sep; 10:82
• Contact Point Address:
  Lidwine B Mokkink, Department of Epidemiology and Biostatistics and the EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
• Email:
  w.mokkink@vumc.nl
• Federal Fiscal Year:
  2010
• Performing Organization:
  University of Washington
• Peer Reviewed:
  True
• Start Date:
  20050701
• Source Full Name:
  BMC Medical Research Methodology
• End Date:
  20250630
• Collection(s):
  National Institute for Occupational Safety and Health
• Main Document Checksum:
  urn:sha-512:483e4c8d5c6b4ef22b26abe5265532c6de279c1ce75565d99a863ab06428e35a02f397beba0ee03ff5a6bd31830d6c090bcf2aea22d6e81f4f31b103ebc96166
• Download URL:
  https://stacks.cdc.gov/view/cdc/214464/cdc_214464_DS1.pdf
• File Type:
  [PDF - 284.76 KB ]