Science Forum: Consensus-based guidance for conducting and reporting multi-analyst studies

B. Aczel; B. Szaszi; G. Nilsonne; O.R. van den Akker; C.J. Albers; M.A. van Assen; J.A. Bastiaansen; D. Benjamin; U. Boehm; R. Botvinik-Nezer; L.F. Bringmann; N.A. Busch; E. Caruyer; A.M. Cataldo; N. Cowan; A. Delios; N.N.N. van Dongen; C. Donkin; J.B. van Doorn; A. Dreber; G. Dutilh; G.F. Egan; M.A. Gernsbacher; R. Hoekstra; S. Hoffmann; F. Holzmeister; J. Huber; M. Johannesson; K.J. Jonas; A.T. Kindel; M. Kirchler; Y.K. Kunkels; D.S. Lindsay; J.F. Mangin; D. Matzke; M.R. Munafo; B.R. Newell; B.A. Nosek; R.A. Poldrack; D. van Ravenzwaaij; J. Rieskamp; M.J. Salganik; A. Sarafoglou; T. Schonberg; M. Schweinsberg; D. Shanks; R. Silberzahn; D.J. Simons; B.A. Spellman; S. St-Jean

doi:10.7554/eLife.72185

Science Forum: Consensus-based guidance for conducting and reporting multi-analyst studies

B. Aczel^*, B. Szaszi^*, G. Nilsonne, O.R. van den Akker, C.J. Albers, M.A. van Assen, J.A. Bastiaansen, D. Benjamin, U. Boehm, R. Botvinik-Nezer, L.F. Bringmann, N.A. Busch, E. Caruyer, A.M. Cataldo, N. Cowan, A. Delios, N.N.N. van Dongen, C. Donkin, J.B. van Doorn, A. DreberG. Dutilh, G.F. Egan, M.A. Gernsbacher, R. Hoekstra, S. Hoffmann, F. Holzmeister, J. Huber, M. Johannesson, K.J. Jonas, A.T. Kindel, M. Kirchler, Y.K. Kunkels, D.S. Lindsay, J.F. Mangin, D. Matzke, M.R. Munafo, B.R. Newell, B.A. Nosek, R.A. Poldrack, D. van Ravenzwaaij, J. Rieskamp, M.J. Salganik, A. Sarafoglou, T. Schonberg, M. Schweinsberg, D. Shanks, R. Silberzahn, D.J. Simons, B.A. Spellman, S. St-Jean

^*Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Original language	English
Article number	e72185
Number of pages	13
Journal	Elife
Volume	10
DOIs	https://doi.org/10.7554/eLife.72185
Publication status	Published - 9 Nov 2021

Keywords

multi-analyst
metascience
statistical practice
science forum
expert consensus
analytical variability
None
REPRODUCIBILITY
DELPHI

Access to Document

10.7554/eLife.72185Licence: CC BY

Cite this

Aczel, B., Szaszi, B., Nilsonne, G., van den Akker, O. R., Albers, C. J., van Assen, M. A., Bastiaansen, J. A., Benjamin, D., Boehm, U., Botvinik-Nezer, R., Bringmann, L. F., Busch, N. A., Caruyer, E., Cataldo, A. M., Cowan, N., Delios, A., van Dongen, N. N. N., Donkin, C., van Doorn, J. B., ... St-Jean, S. (2021). Science Forum: Consensus-based guidance for conducting and reporting multi-analyst studies. Elife, 10, Article e72185. https://doi.org/10.7554/eLife.72185

@article{d70b5d9a1c924b8998ed584d72e37340,

title = "Science Forum: Consensus-based guidance for conducting and reporting multi-analyst studies",

abstract = "Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.",

keywords = "multi-analyst, metascience, statistical practice, science forum, expert consensus, analytical variability, None, REPRODUCIBILITY, DELPHI",

author = "B. Aczel and B. Szaszi and G. Nilsonne and {van den Akker}, O.R. and C.J. Albers and {van Assen}, M.A. and J.A. Bastiaansen and D. Benjamin and U. Boehm and R. Botvinik-Nezer and L.F. Bringmann and N.A. Busch and E. Caruyer and A.M. Cataldo and N. Cowan and A. Delios and {van Dongen}, N.N.N. and C. Donkin and {van Doorn}, J.B. and A. Dreber and G. Dutilh and G.F. Egan and M.A. Gernsbacher and R. Hoekstra and S. Hoffmann and F. Holzmeister and J. Huber and M. Johannesson and K.J. Jonas and A.T. Kindel and M. Kirchler and Y.K. Kunkels and D.S. Lindsay and J.F. Mangin and D. Matzke and M.R. Munafo and B.R. Newell and B.A. Nosek and R.A. Poldrack and {van Ravenzwaaij}, D. and J. Rieskamp and M.J. Salganik and A. Sarafoglou and T. Schonberg and M. Schweinsberg and D. Shanks and R. Silberzahn and D.J. Simons and B.A. Spellman and S. St-Jean",

note = "Funding Information: VENI fellowship grant Publisher Copyright: {\textcopyright} Aczel et al.",

year = "2021",

month = nov,

day = "9",

doi = "10.7554/eLife.72185",

language = "English",

volume = "10",

journal = "Elife",

issn = "2050-084X",

publisher = "eLife Sciences Publications, Ltd",

}

Aczel, B, Szaszi, B, Nilsonne, G, van den Akker, OR, Albers, CJ, van Assen, MA, Bastiaansen, JA, Benjamin, D, Boehm, U, Botvinik-Nezer, R, Bringmann, LF, Busch, NA, Caruyer, E, Cataldo, AM, Cowan, N, Delios, A, van Dongen, NNN, Donkin, C, van Doorn, JB, Dreber, A, Dutilh, G, Egan, GF, Gernsbacher, MA, Hoekstra, R, Hoffmann, S, Holzmeister, F, Huber, J, Johannesson, M, Jonas, KJ, Kindel, AT, Kirchler, M, Kunkels, YK, Lindsay, DS, Mangin, JF, Matzke, D, Munafo, MR, Newell, BR, Nosek, BA, Poldrack, RA, van Ravenzwaaij, D, Rieskamp, J, Salganik, MJ, Sarafoglou, A, Schonberg, T, Schweinsberg, M, Shanks, D, Silberzahn, R, Simons, DJ, Spellman, BA & St-Jean, S 2021, 'Science Forum: Consensus-based guidance for conducting and reporting multi-analyst studies', Elife, vol. 10, e72185. https://doi.org/10.7554/eLife.72185

TY - JOUR

T1 - Science Forum

T2 - Consensus-based guidance for conducting and reporting multi-analyst studies

AU - Aczel, B.

AU - Szaszi, B.

AU - Nilsonne, G.

AU - van den Akker, O.R.

AU - Albers, C.J.

AU - van Assen, M.A.

AU - Bastiaansen, J.A.

AU - Benjamin, D.

AU - Boehm, U.

AU - Botvinik-Nezer, R.

AU - Bringmann, L.F.

AU - Busch, N.A.

AU - Caruyer, E.

AU - Cataldo, A.M.

AU - Cowan, N.

AU - Delios, A.

AU - van Dongen, N.N.N.

AU - Donkin, C.

AU - van Doorn, J.B.

AU - Dreber, A.

AU - Dutilh, G.

AU - Egan, G.F.

AU - Gernsbacher, M.A.

AU - Hoekstra, R.

AU - Hoffmann, S.

AU - Holzmeister, F.

AU - Huber, J.

AU - Johannesson, M.

AU - Jonas, K.J.

AU - Kindel, A.T.

AU - Kirchler, M.

AU - Kunkels, Y.K.

AU - Lindsay, D.S.

AU - Mangin, J.F.

AU - Matzke, D.

AU - Munafo, M.R.

AU - Newell, B.R.

AU - Nosek, B.A.

AU - Poldrack, R.A.

AU - van Ravenzwaaij, D.

AU - Rieskamp, J.

AU - Salganik, M.J.

AU - Sarafoglou, A.

AU - Schonberg, T.

AU - Schweinsberg, M.

AU - Shanks, D.

AU - Silberzahn, R.

AU - Simons, D.J.

AU - Spellman, B.A.

AU - St-Jean, S.

PY - 2021/11/9

Y1 - 2021/11/9

N2 - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

AB - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

KW - multi-analyst

KW - metascience

KW - statistical practice

KW - science forum

KW - expert consensus

KW - analytical variability

KW - None

KW - REPRODUCIBILITY

KW - DELPHI

U2 - 10.7554/eLife.72185

DO - 10.7554/eLife.72185

M3 - Article

SN - 2050-084X

VL - 10

JO - Elife

JF - Elife

M1 - e72185

ER -