Converging the Infinite Limits of Qualitative and Quantitative Research: Inter and Intra Rater Reliability Metrics as Measures of Evidence Strength in Systematic Reviews of the Literature
Location
1053
Format Type
Paper
Format Type
Paper
Start Date
January 2016
End Date
January 2016
Abstract
Despite the call from highly reputable research organizations for evidence-based educational research, the use of a rater agreement procedure to determine the strength of the literature screening procedures employed in a systematic review of the literature has yet to be considered by educational researchers. The U.S. Department of Education’s Institute of Education Sciences (IES) has led the effort to critically assess the scientific literature for the purpose of reporting to stakeholders the effectiveness of educational interventions. Similar to the work conducted by the Cochrane and Campbell Collaboration, the What Works Clearinghouse (WWC), a subdivision of the IES, is responsible for conducting and reporting systematic reviews of the literature through the use of a protocol published in the What Works Clearinghouse Procedures and Standards Handbook Version 3.0. (2014). The handbook calls for reporting statistical significance, effect sizes, and an improvement index. However, there is no indication of the level of agreement, the reliability of the literature screening procedures, and the review of the eligible studies against the WWC standards. To measure the consistency of the appraisal steps in a systematic review of the literature, this paper proposes the use of measures of inter-rater reliability and inter-rater agreement. Specifically, we explore the conceptual differences between inter-rater reliability and inter-rater agreement metrics, compare the use of several statistical procedures for calculating these metrics and provide examples on how to use these measures as indicators of the strength of the evidence presented in a systematic review of the literature.
Converging the Infinite Limits of Qualitative and Quantitative Research: Inter and Intra Rater Reliability Metrics as Measures of Evidence Strength in Systematic Reviews of the Literature
1053
Despite the call from highly reputable research organizations for evidence-based educational research, the use of a rater agreement procedure to determine the strength of the literature screening procedures employed in a systematic review of the literature has yet to be considered by educational researchers. The U.S. Department of Education’s Institute of Education Sciences (IES) has led the effort to critically assess the scientific literature for the purpose of reporting to stakeholders the effectiveness of educational interventions. Similar to the work conducted by the Cochrane and Campbell Collaboration, the What Works Clearinghouse (WWC), a subdivision of the IES, is responsible for conducting and reporting systematic reviews of the literature through the use of a protocol published in the What Works Clearinghouse Procedures and Standards Handbook Version 3.0. (2014). The handbook calls for reporting statistical significance, effect sizes, and an improvement index. However, there is no indication of the level of agreement, the reliability of the literature screening procedures, and the review of the eligible studies against the WWC standards. To measure the consistency of the appraisal steps in a systematic review of the literature, this paper proposes the use of measures of inter-rater reliability and inter-rater agreement. Specifically, we explore the conceptual differences between inter-rater reliability and inter-rater agreement metrics, compare the use of several statistical procedures for calculating these metrics and provide examples on how to use these measures as indicators of the strength of the evidence presented in a systematic review of the literature.
Comments
Breakout Session B