ReproducibiliTea Blog

Errors in Research

“Fallibility in Science- Responding to Errors in the Work of Oneself and Others”

This was the first session of year 2022 and revolved around a paper discussion on Errors in Research. It was led by Laura Klinkhamer, a PHD student at The University of Edinburgh. Her research interests lie at the intersection of neuroscience and psychology. The discussion was on Professor Dorothy Bishop’s 2018 commentary paper ‘Fallibility in Science: Responding to Errors in the Work of Oneself and Others’. Apart from the paper discussion, the session involved interactive sessions with anonymous polls on Mentimeter.com and some interesting discussions in the breakout rooms. 

The session began with imagining a scenario where a PHD student runs a series of studies to find a positive effect. After getting null findings in three studies, the student changed the design and found a statistically significant effect in the fourth study. This resulted in paper publication in a prestigious journal with student as first author. The study was also featured on National Public Radio. However, after two weeks the student realized as a consequence of preparing for a conference talk that the groups in the study were miscoded and the study was a faulty one. The same scenario was asked to be imagined by the participants in the session and to report their answers anonymously on Mentimeter.com. 

According to Azoulay, Bonatti and Krieger (2017), there was an average decline of 10% in subsequent citations of early work of authors who publicly admitted their mistake. However, the effect was small when the mistake made was an honest one. Moreover, there was no reputational damage in case of junior researchers. According to Hosseini, Hillhorst, de Beaufort & Fanelli (2018), 14 authors who self-retracted their papers believed their reputation would be damaged badly. However, in reality, self-retraction did not damage their reputation but improved it. 

Incentives for Errors in Research or Research Misconduct:

  1. Pressure from colleagues, institutions and journal editors to publish more and more papers
  2. Progression in academic career is determined greatly by metrics that incentivize publications and not retractions

Unfortunately, according to Bishop (2018) there are very few incentives for honesty in academic careers. Participants were encouraged to share their opinions on Mentimeter.com on what would they do to incentivize scientific integrity. 

Open Research:

  1. Research that is publicly accessible does not indicate that it is free from errors. However, open data and open code enhances the chances of error detection by the other authors
  2. Open research encourages scientists to double check their data and code before publication
  3. Open research helps normalize error detections and reduces stigma which eventually leads to scientific accuracy 

How to Respond to Errors in the Work of Other Researchers:

There are different platforms to do that including-

  • Contacting researchers directly
  • Contacting researchers via journal (if possible)
  • Preprint servers
  • PubMed Commons (discontinued)
  • PubPeer (commentators can be anonymous)
  • Twitter
  • Personal blogs
  • OSF and Octopus (emerging platforms)

One of the drawbacks of anonymous platforms is that they often result in criticism of someone’s work that can be harsh and discouraging. When responding to errors in the work of other scientists it is important to make no assumptions. Because a failure to replicate an original study can be due to reasons beyond incompetence or fraudulent intentions. The scale of error can be useful while approaching the situation.

Scale of errors:

  • Honest errors- coding mistakes
  • Paltering- using a truthful statement to mislead by failing to provide the relevant contextual information
  • P-hacking
  • Citing only a part of literature that matches with one’s position. Commonly referred to as confirmation bias
  • Inaccurate presentation of results from cited studies
  • Inventing fake data
  • Paper mills- businesses producing fake studies for profits

There was a little discussion on the case of Diederik Stapel who was fired instantly after it was discovered that he faked a large-scale data during his academic career. Moreover, some discussion was done on paper mills that are polluting the scientific literature for profits. An important question remains: who are/should be responsible for detecting and responding to large errors? 

  1. At an internal level, head of the department/lab, whistleblowing policy and research misconduct policy
  2. Journals 
  3. Separate institutes like UKRIO (UK Research Integrity Office
  4. Technology
  5. External researchers

There was a lot more to be discussed and hopefully the discussion can continue in later discussions and/or the conference. There is a ‘Edinburgh Open Research Conference’ on Friday 27 May, 2022 organised by the Library Research Support Team and EORI/Edinburgh ReproducibiliTea. SAVE THE DATE!!!!

Anonymous responses the participants on Mentimeter.com:

This blog is written by Sumbul Syed

SOCIALS:

Edinburgh RT Twitter

Edinburgh RT OSF page

Edinburgh RT mailing list

For any questions/suggestions, please send us an email at edinburgh.reproducibilitea@ed.ac.uk

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s