In our first Edinburgh ReproducibiliTea session of 2023, Dr Roman Briker gave a talk on pre-registration and Registered Reports. Dr Briker is an Assistant Professor in Organisational Behaviour at Maastricht University School of Business and Economics, and an Open Science Ambassador at the School of Business and Economics.
Reproducibility crisis and questionable research practices
Many academic journals are interested in significant results, those with a P-value of less than 0.05. Dr Briker shared a personal experience of writing his first paper and spotting an error in his draft that would impact the results of his statistical analyses. He was worried that correcting the error would lead to his findings being non-significant, and that journals would no longer be interested in his work.
This led Dr Briker to realise that this is not the way research should work. In his talk, Dr Briker suggested that this current model of academic publishing – the culture of “publish or perish” – contributes to the reproducibility crisis, as significant results are published and non-significant results are filed away. Dr Briker also gave examples of scientific fraud, including Dan Ariely and Daryl Bem, and reports from a survey which suggested that 8% of Dutch scientists have at some point falsified data. Overall, the focus on significant outcomes reduces our focus on rigorous methodology.
Dr Briker mentioned that the issue of irreproducibility impacts all fields of research, and that only 25% to 60% of scientific findings are replicable. He spoke about questionable research practices which have been allowed to thrive in our current research culture, including HARKing (Hypothesising After Results are Known), selective reporting, optimal/selective stopping of experiments, changing control variables, playing around with outliers, changing the inclusion or exclusion criteria, using different analytical methods, and rounding off P-values (e.g. reporting a P value of 0.53 as P = < 0.5).
Pre-registration and registered reports
Dr Briker suggested pre-registration and Registered Reports as potential solutions to these problems.
A pre-registration is a publicly time-stamped pre-specification of a research study design, including hypotheses, required sample sizes, exclusion criteria, and planned analyses. It is completed prior to data collection and is not peer-reviewed (Logg & Dorison, 2021).
A Registered Report goes further than a pre-registration, including the introduction, theory and hypothesis, proposed methods and analyses (Chambers & Tzavella, 2022). This is submitted to a journal, or platform such as Peer Community In Registered Reports, for peer-review prior to data collection. Once the Registered Report is approved by reviewers, it gains in-principle acceptance for publication in a journal, and the results will be published whether they are significant or not, as long as the plan outlined in the Registered Report is followed.
In his talk, Dr Briker explained what parts of a study design should be pre-registered, and gave an example of his own pre-registration. He also highlighted a number of templates available, and busted some myths surrounding concerns researchers may have about pre-registering a study.
Slides, references and pre-registration templates mentioned in Dr Briker’s talk are available on OSF: https://osf.io/cgkua/
The session recording is available on our YouTube channel.
This blog is written by Emma Wilson
SOCIALS:
- Edinburgh RT Twitter
- Edinburgh RT OSF page
- Edinburgh RT mailing list
For any questions/suggestions, please send us an email at edinburgh.reproducibilitea@ed.ac.uk
Edinburgh Open Research Initiative 