Converging the Infinite Limits of Qualitative and Quantitative Research: Inter and Intra Rater Reliability Metrics as Measures of Evidence Strength in Systematic Reviews of the Literature

Location

1053

Format Type

Paper

Format Type

Paper

Start Date

January 2016

End Date

January 2016

Abstract

Despite the call from highly reputable research organizations for evidence-based educational research, the use of a rater agreement procedure to determine the strength of the literature screening procedures employed in a systematic review of the literature has yet to be considered by educational researchers. The U.S. Department of Education’s Institute of Education Sciences (IES) has led the effort to critically assess the scientific literature for the purpose of reporting to stakeholders the effectiveness of educational interventions. Similar to the work conducted by the Cochrane and Campbell Collaboration, the What Works Clearinghouse (WWC), a subdivision of the IES, is responsible for conducting and reporting systematic reviews of the literature through the use of a protocol published in the What Works Clearinghouse Procedures and Standards Handbook Version 3.0. (2014). The handbook calls for reporting statistical significance, effect sizes, and an improvement index. However, there is no indication of the level of agreement, the reliability of the literature screening procedures, and the review of the eligible studies against the WWC standards. To measure the consistency of the appraisal steps in a systematic review of the literature, this paper proposes the use of measures of inter-rater reliability and inter-rater agreement. Specifically, we explore the conceptual differences between inter-rater reliability and inter-rater agreement metrics, compare the use of several statistical procedures for calculating these metrics and provide examples on how to use these measures as indicators of the strength of the evidence presented in a systematic review of the literature.

Comments

Breakout Session B

This document is currently not available here.

Share

COinS
 
Jan 14th, 5:00 PM Jan 14th, 5:20 PM

Converging the Infinite Limits of Qualitative and Quantitative Research: Inter and Intra Rater Reliability Metrics as Measures of Evidence Strength in Systematic Reviews of the Literature

1053

Despite the call from highly reputable research organizations for evidence-based educational research, the use of a rater agreement procedure to determine the strength of the literature screening procedures employed in a systematic review of the literature has yet to be considered by educational researchers. The U.S. Department of Education’s Institute of Education Sciences (IES) has led the effort to critically assess the scientific literature for the purpose of reporting to stakeholders the effectiveness of educational interventions. Similar to the work conducted by the Cochrane and Campbell Collaboration, the What Works Clearinghouse (WWC), a subdivision of the IES, is responsible for conducting and reporting systematic reviews of the literature through the use of a protocol published in the What Works Clearinghouse Procedures and Standards Handbook Version 3.0. (2014). The handbook calls for reporting statistical significance, effect sizes, and an improvement index. However, there is no indication of the level of agreement, the reliability of the literature screening procedures, and the review of the eligible studies against the WWC standards. To measure the consistency of the appraisal steps in a systematic review of the literature, this paper proposes the use of measures of inter-rater reliability and inter-rater agreement. Specifically, we explore the conceptual differences between inter-rater reliability and inter-rater agreement metrics, compare the use of several statistical procedures for calculating these metrics and provide examples on how to use these measures as indicators of the strength of the evidence presented in a systematic review of the literature.