Faculty Articles

Title

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Author(s)

Charles R. Ebersole, University of VirginiaFollow
Maya B. Mathur, Stanford UniversityFollow
Erica Baranski, University of HoustonFollow
Diane-Jo Bart-Plange, University of VirginiaFollow
Nicholas R. Buttrick, University of VirginiaFollow
Christopher R. Chartier, Ashland UniversityFollow
Katherine S. Corker, Grand Valley State UniversityFollow
Martin Corley, The University of EdinburghFollow
Joshua K. Hartshorne, Boston CollegeFollow
Hans Rocha Ijzerman, Université Grenoble AlpesFollow
Ljiljana B. Lazarević, University of BelgradeFollow
Hugh Rabagliati, The University of EdinburghFollow
Ivan Ropovik, University of PresovFollow
Balazs Aczel, Eötvös Loránd UniversityFollow
Lena F. Aeschbach, University of BaselFollow
Luca Andrighetto, University of GenovaFollow
Jack D. Arnal, McDaniel CollegeFollow
Holly Arrow, University of OregonFollow
Peter Babincak, University of PresovFollow
Bence E. Bakos, Eötvös Loránd UniversityFollow
Gabriel Banik, University of PresovFollow
Ernest Baskin, Saint Joseph’s University
Radomir Belopavlović, University of Novi SadFollow
Michael H. Bernstein, Brown UniversityFollow
Michał Białek, University of WroclawFollow
Nicholas G. Bloxsom, Ashland University
Bojana Bodroža, University of Novi SadFollow
Diane B. V. Bonfiglio, Ashland UniversityFollow
Leanne Boucher, Nova Southeastern UniversityFollow
Florian Brühlmann, University of BaselFollow
Claudia C. Brumbaugh, City University of New YorkFollow
Erica Casini, University of Milano-BicoccaFollow
Yiling Chen, Harvard UniversityFollow
Carlo Chiorri, University of GenovaFollow
William J. Chopik, Michigan State UniversityFollow
Oliver Christ, FernUniversität in HagenFollow
Antonia M. Ciunci, University of Rhode Island
Heather M. Claypool, Miami UniversityFollow
Sean Coary, Loyola University ChicagoFollow
Marija V. Čolić, University of BelgradeFollow
W. Matthew Collins, Nova Southeastern UniversityFollow
Paul R. Curran, Grand Valley State UniversityFollow
Chris R. Day, Coventry University
Benjamin Dering, University of StirlingFollow
Anna Dreber, Stockholm School of EconomicsFollow
John E. Edlund, Rochester Institute of TechnologyFollow
Filipe Falcão, University of Porto
Anna Fedor, Centre for Ecological Research, Budapest, HungaryFollow
Lily Feinberg, Boston College
Ian R. Ferguson, Virginia Commonwealth UniversityFollow
Máire Ford, Loyola Marymount UniversityFollow
Michael C. Frank, Stanford UniversityFollow
Emily Fryberger, Pacific Lutheran University
Alexander Garinther, University of OregonFollow
Katarzyna Gawryluk, Kozminski UniversityFollow
Kayla Ashbaugh, Rose-Hulman Institute of Technology
Mauro Giacomantonio, Sapienza University of RomeFollow
Steffen R. Giessner, Erasmus UniversityFollow
Jon E. Grahe, Pacific Lutheran UniversityFollow
Rosanna E. Guadagno, Stanford UniversityFollow
Ewa Hałasa, Maria Curie-Skłodowska University
Peter J. B. Hancock, University of StirlingFollow
Rias A. Hilliard, Rose-Hulman Institute of Technology
Joachim Hüffmeier, TU Dortmund UniversityFollow
Sean Hughes, Ghent UniversityFollow
Katarzyna Idzikowska, Kozminski UniversityFollow
Michael Inzlicht, University of TorontoFollow
Alan Jern, Rose-Hulman Institute of TechnologyFollow
William Jiménez-Leal, Universidad de los AndesFollow
Magnus Johannesson, Stockholm School of EconomicsFollow
Jennifer A. Joy-Gaba, Virginia Commonwealth UniversityFollow
Mathias Kauff, Medical School HamburgFollow
Danielle J. Kellier, University of Pennsylvania
Grecia Kessinger, Brigham Young University–Idaho
Mallory C. Kidwell, University of UtahFollow
Amanda M. Kimbrough, University of Texas at Dallas
Josiah P. J. King, University of EdinburghFollow
Vanessa S. Kolb, University of Rhode IslandFollow
Sabina Kołodziej, Kozminski UniversityFollow
Marton Kovacs, ELTE Eötvös Loránd UniversityFollow
Karolina Krasuska, Maria Curie-Skłodowska UniversityFollow
Sue Kraus, Fort Lewis CollegeFollow
Lacy E. Krueger, Texas A&M University-CommerceFollow
Katarzyna Kuchno, Maria Curie-Skłodowska University
Caio Ambrosio Lage, Pontifical Catholic University of Rio de Janeiro
Eleanor V. Langford, University of Virginia
Carmel A. Levitan, Occidental CollegeFollow
Tiago Jessé Souza de Lima, University of Brasília
Hause Lin, University of TorontoFollow
Samuel Lins, University of PortoFollow
Jia E. Loy, University of EdinburghFollow
Dylan Manfredi, University of PennsylvaniaFollow
Łukasz Markiewicz, Kozminski UniversityFollow
Madhavi Menon, Nova Southeastern UniversityFollow
Brett Mercier, University of California, IrvineFollow
Mitchell Metzger, Ashland UniversityFollow
Venus Meyet, Brigham Young University–Idaho
Ailsa E. Millen, University of StirlingFollow
Jeremy K. Miller, Willamette UniversityFollow
Andres Montealegre, Cornell UniversityFollow
Don A. Moore, University of California - BerkeleyFollow
Rafał Muda, Maria Curie-Skłodowska UniversityFollow
Gideon Nave, University of PennsylvaniaFollow
Austin Lee Nichols, Duy Tan UniversityFollow
Sarah A. Novak, Hofstra UniversityFollow
Christian Nunnally, Rose-Hulman Institute of Technology
Ana Orlić, University of BelgradeFollow
Anna Palinkas, ELTE Eötvös Loránd University
Angelo Panno, European University of RomeFollow
Kimberly P. Parks, University of Virginia
Ivana Pedović, University of NišFollow
Emilian Pękala, Maria Curie-Skłodowska University
Matthew R. Penner, Cornell University
Sebastiaan Pessers, University of LeuvenFollow
Boban Petrović, University of BelgradeFollow
Thomas Pfeiffer, Massey UniversityFollow
Damian Pieńkosz, Maria Curie-Skłodowska University
Emanuele Preti, University of Milano-BicoccaFollow
Danka Purić, University of BelgradeFollow
Tiago Ramos, University of Porto
Jonathan Ravid, Boston College
Timothy S. Razza, Nova Southeastern UniversityFollow
Katrin Rentzsch, Psychologische Hochschule BerlinFollow
Juliette Richetin, University of Milano-BicoccaFollow
Sean C. Rife, Murray State UniversityFollow
Anna Dalla Rosa, University of PadovaFollow
Kaylis Hase Rudy, Brigham Young University–Idaho
Janos Salamon, ELTE Eötvös Loránd University
Blair Saunders, University of DundeeFollow
‪Przemysław Sawicki‬, Kozminski UniversityFollow
Kathleen Schmidt, Southern Illinois University CarbondaleFollow
Kurt Schuepfer, Miami University
Thomas Schultze, University of GöttingenFollow
Stefan Schulz-Hardt, University of GöttingenFollow
Astrid Schütz, University of BambergFollow
Ani N. Shabazian, Loyola Marymount UniversityFollow
Rachel L. Shubella, Rose-Hulman Institute of Technology
Adam Siegel, Cultivate Labs, Chicago, Illinois
Rúben Silva, University of PortoFollow
Barbara Sioma, Maria Curie-Skłodowska UniversityFollow
Lauren Skorb, Boston College
Luana Elayne Cunha de Souza, Universidade de Fortaleza
Sara Steegen, University of LeuvenFollow
L A. R. Stein, Brown UniversityFollow
R Weylin Sternglanz, Nova Southeastern UniversityFollow
Darko Stojilović, Belgrade, SerbiaFollow
Daniel Storage, University of DenverFollow
Gavin Brent Sullivan, Coventry UniversityFollow
Barnabas Szaszi, ELTE Eötvös Loránd UniversityFollow
Peter Szecsi, ELTE Eötvös Loránd UniversityFollow
Orsolya Szöke, ELTE Eötvös Loránd UniversityFollow
Attila Szuts, ELTE Eötvös Loránd UniversityFollow
Manuela Thomae, MEU - Die Multiversität
Natasha D. Tidwell, Fort Lewis CollegeFollow
Carly Tocco, City University of New York
Ann-Kathrin Torka, TU Dortmund UniversityFollow
Francis Tuerlinckx, University of LeuvenFollow
Wolf Vanpaemel, University of LeuvenFollow
Leigh Ann Vaughn, Ithaca CollegeFollow
Michelangelo Vianello, University of PadovaFollow
Domenico Viganola, Stockholm School of EconomicsFollow
Maria Vlachou, University of LeuvenFollow
Ryan J. Walker, Miami University
Sophia Christin Weissgerber, University of KasselFollow
Aaron L. Wichman, Western Kentucky UniversityFollow
Bradford J. Wiggins, Brigham Young University–IdahoFollow
Daniel Wolf, University of Bamberg
Michael J. Wood, University of WinchesterFollow
David Zealley, Brigham Young University–Idaho
Iris Žeželj, University of BelgradeFollow
Mark Zrubka, University of Amsterdam
Brian A. Nosek, Center for Open Science, Charlottesville, VirginiaFollow

Document Type

Article

Publication Date

2020

Publication Title

Advances in Methods and Practices in Psychological Science

ISSN or ISBN

2515-2467

Volume

3

Issue/Number

3

Abstract/Excerpt

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δr = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

DOI

https://dx.doi.org/10.1177/2515245920958687

ORCID ID

0000-0002-8607-2579, 0000-0003-1629-3699, 0000-0001-6025-7068, 0000-0001-8840-4285

Peer Reviewed

Find in your library

Share

COinS