EORI Bulletin

13/05/2022 5-minute update    

EORI keeps an eye on changes in the fields of Open Science, FAIR data principles, and others, and directs any interested parties to important updates:   

  • Last month, Vergoulis et al. (2022) launched the beta of BIP! Scholar. This is an online service that allows researchers to set up their own academic profile in accordance with Open Science guidelines for fair research assessment. One of the goals of BIP! Scholar is to curb the overreliance on performance indicators such as the h-index, which may be an inaccurate reflection of a researchers’ academic experiences. You can read more about the platform here and sign up for the beta here.  
  • Here’s an interesting article by van der Wal et al. (2022) on the merits and drawbacks of publishing academic talks online. As the COVID-19 pandemic has caused a massive shift towards online conferences, this has led to increased accessibility and reach for academic talks. However, as van der Wal et al. argue, this comes with certain ethical considerations, including data privacy, and it may also make the speaker open to persistent criticism, which could be challenging for early career researchers. Van der Wal et al. argue that the speaker should decide whether their talk is made available online. Furthermore, talks may need to be edited before they are put online to avoid ethical problems.  
  • On the topic of academic feedback, here is another interesting article by Iborra et al. (2022) on how to give constructive criticism on preprints using the FAST (Focused, Appropriate, Specific and Transparent) principle.  
  • And once again, don’t forget to sign up for Edinburgh University’s first Open Research conference on the 27th of May. It’s free to all students and staff at the University of Edinburgh, and events will be held both online and in person. The conference will feature talks and workshops on how to get started with Open Science, practical considerations in Open Science, what resources are available at the University of Edinburgh, as well as many more.  

The best way to get more updates is to follow EORI on Twitter.   

EORI Bulletin

29/04/2022 5-minute update   

EORI keeps an eye on changes in the fields of Open Science, FAIR data principles, and others, and directs any interested parties to important updates:  

  • Smith and Sandbrik published this interesting paper on some of the potential ethical issues with Open Science research practices in biological research.  They posit that preregistration could help encourage risk assessment in the earlier stages of the research lifecycle and call for responsible and considered data sharing and access. Here’s also a WIRED article on their paper. 
  • Here’s a great introductory resource for data visualisation with R. It’s aimed at researchers who have not used R before and features many different types of graphs and example code.  
  • And finally, don’t forget to sign up for Edinburgh University’s first Open Research conference on the 27th of May. It’s free to all students and staff at the University of Edinburgh, and events will be held both online and in person. The conference will feature talks and workshops on how to get started with Open Science, practical considerations in Open Science, what resources are available at the University of Edinburgh, as well as many more.  

The best way to get more updates is to follow EORI on Twitter.  

EORI Bulletin

18/03/2022 5-minute update  

EORI keeps an eye on changes in the fields of Open Science, FAIR data principles, and others, and directs any interested parties to important updates: 

  • The Arqus European University Alliance has joined a steadily increasing number of research institutions that are committing to Open Science principles. This is a great step towards removing accessibility barriers and making scientific research easily accessible within and beyond Europe. You can read their Openness position paper here.  
  • A lawsuit in which ResearchGate was sued for hosting 50 copyrighted papers has ended inconclusively for both sides. Though ResearchGate was ruled responsible for hosting the papers, the status of any other paper hosted on the platform that may infringe with copyright law remains unclear. Still, this is a worrying precedent for the future of openly accessible research, and ResearchGate wants to appeal the decision. You can read more about the case here
  • And finally, the University of Surrey has launched its Open Research page, which hosts many valuable resources and guidelines. You can find them here

The best way to get more updates is to follow EORI on Twitter

EORI Bulletin

04/03/2022 – 5-minute update   

EORI keeps an eye on changes in the fields of Open Science, FAIR data principles, and others, and directs any interested parties to important updates:   

  • Open Science scholarship has revolutionised the scientific community, yet the sheer number of novel terms and concepts associated with it may be daunting for new researchers. To reduce entry barriers to open scholarship, the FORRT project has developed a community-sourced glossary of 250 relevant open scholarship terms. This is a great resource if you have ever wondered what PARKing is or what Type M errors are, and you can learn more about how the project came to be here
  • Gopalakrishna et al. (2022) published an investigation into the prevalence of questionable research practices and academic misconduct in research circles in the Netherlands. Their findings are worrying, with at least one in two researchers reporting that they frequently participate in questionable research practices, whilst one in twelve reported falsifying or fabricating their data at least once. Gopalakrishna and colleagues suggest that reducing the “publish or perish” mentality and amplifying the role of the peer reviewer in “gatekeeping” research quality and integrity may help reduce the widespread use of questionable research practices. 
  • This preprint by Steve Haroz offers a comprehensive breakdown of differences between five preregistration platforms (GitHub, AsPredicted, Zenodo, OSF (template) & OSF (open-ended)). This can help researchers make informed decisions when deciding where to preregister their study, but also highlights what information is especially vital to include in a preregistered report.  

The best way to get more updates is to follow EORI on Twitter

EORI Bulletin

18/02/2022 – 5-minute update  

EORI keeps an eye on changes in the fields of Open Science, FAIR data principles, and others, and directs any interested parties to important updates:  

  • A recent study by Skiles and colleagues reported that online conferences are not only good for reducing researchers’ carbon footprint, but also promote diversity and inclusion. The study found that the recent move towards online, rather than in-person, conferences has removed some of the monetary barriers for attendees, especially boosting attendance by women and early career researchers. 
  • In the future, all publicly-funded research conducted in South Africa will be published in open access journals, a draft national open science policy has revealed. The draft aims to promote positive change within the scientific culture and to increase the public benefit of funded research.
  • The Friedrich-Alexander University in Germany has become the first German university to adopt an Open Science policy. With this, the university pledges itself to promoting high-quality, transparent and open access research.  

The best way to get more updates is to follow EORI on Twitter

EORI Bulletin

08/02/2022 – 5-minute update 

After a break, EORI is back for the new year with our 5-minute bulletin! 

EORI keeps an eye on changes in the fields of Open Science, FAIR data principles, and others, and directs any interested parties to important updates: 

  • Recently, NASA launched their Transform to Open Science mission. The program has designated 2023 as the Year of Open Science and aims to use Open Science principles to further accelerate scientific research and to promote the inclusion of historically excluded communities in its science program.  
  • The European University Association published its Open Science Agenda 2025. Its key priority areas are promoting open access, implementing FAIR data practices, and encouraging more responsible research assessment. The purpose of this agenda is to aid its members in the transition towards Open Science.  
  • This article covers the rise of preprints during the COVID-19 pandemic, highlighting both the benefits and pitfalls of rapid data sharing. It suggests that open access data and code sharing are paramount in ensuring the quality of scientific preprints.  

The best way to get more updates is to follow EORI on Twitter

ReproducibiliTea Blog

Errors in Research

“Fallibility in Science- Responding to Errors in the Work of Oneself and Others”

This was the first session of year 2022 and revolved around a paper discussion on Errors in Research. It was led by Laura Klinkhamer, a PHD student at The University of Edinburgh. Her research interests lie at the intersection of neuroscience and psychology. The discussion was on Professor Dorothy Bishop’s 2018 commentary paper ‘Fallibility in Science: Responding to Errors in the Work of Oneself and Others’. Apart from the paper discussion, the session involved interactive sessions with anonymous polls on Mentimeter.com and some interesting discussions in the breakout rooms. 

The session began with imagining a scenario where a PHD student runs a series of studies to find a positive effect. After getting null findings in three studies, the student changed the design and found a statistically significant effect in the fourth study. This resulted in paper publication in a prestigious journal with student as first author. The study was also featured on National Public Radio. However, after two weeks the student realized as a consequence of preparing for a conference talk that the groups in the study were miscoded and the study was a faulty one. The same scenario was asked to be imagined by the participants in the session and to report their answers anonymously on Mentimeter.com. 

According to Azoulay, Bonatti and Krieger (2017), there was an average decline of 10% in subsequent citations of early work of authors who publicly admitted their mistake. However, the effect was small when the mistake made was an honest one. Moreover, there was no reputational damage in case of junior researchers. According to Hosseini, Hillhorst, de Beaufort & Fanelli (2018), 14 authors who self-retracted their papers believed their reputation would be damaged badly. However, in reality, self-retraction did not damage their reputation but improved it. 

Incentives for Errors in Research or Research Misconduct:

  1. Pressure from colleagues, institutions and journal editors to publish more and more papers
  2. Progression in academic career is determined greatly by metrics that incentivize publications and not retractions

Unfortunately, according to Bishop (2018) there are very few incentives for honesty in academic careers. Participants were encouraged to share their opinions on Mentimeter.com on what would they do to incentivize scientific integrity. 

Open Research:

  1. Research that is publicly accessible does not indicate that it is free from errors. However, open data and open code enhances the chances of error detection by the other authors
  2. Open research encourages scientists to double check their data and code before publication
  3. Open research helps normalize error detections and reduces stigma which eventually leads to scientific accuracy 

How to Respond to Errors in the Work of Other Researchers:

There are different platforms to do that including-

  • Contacting researchers directly
  • Contacting researchers via journal (if possible)
  • Preprint servers
  • PubMed Commons (discontinued)
  • PubPeer (commentators can be anonymous)
  • Twitter
  • Personal blogs
  • OSF and Octopus (emerging platforms)

One of the drawbacks of anonymous platforms is that they often result in criticism of someone’s work that can be harsh and discouraging. When responding to errors in the work of other scientists it is important to make no assumptions. Because a failure to replicate an original study can be due to reasons beyond incompetence or fraudulent intentions. The scale of error can be useful while approaching the situation.

Scale of errors:

  • Honest errors- coding mistakes
  • Paltering- using a truthful statement to mislead by failing to provide the relevant contextual information
  • P-hacking
  • Citing only a part of literature that matches with one’s position. Commonly referred to as confirmation bias
  • Inaccurate presentation of results from cited studies
  • Inventing fake data
  • Paper mills- businesses producing fake studies for profits

There was a little discussion on the case of Diederik Stapel who was fired instantly after it was discovered that he faked a large-scale data during his academic career. Moreover, some discussion was done on paper mills that are polluting the scientific literature for profits. An important question remains: who are/should be responsible for detecting and responding to large errors? 

  1. At an internal level, head of the department/lab, whistleblowing policy and research misconduct policy
  2. Journals 
  3. Separate institutes like UKRIO (UK Research Integrity Office
  4. Technology
  5. External researchers

There was a lot more to be discussed and hopefully the discussion can continue in later discussions and/or the conference. There is a ‘Edinburgh Open Research Conference’ on Friday 27 May, 2022 organised by the Library Research Support Team and EORI/Edinburgh ReproducibiliTea. SAVE THE DATE!!!!

Anonymous responses the participants on Mentimeter.com:

This blog is written by Sumbul Syed

SOCIALS:

Edinburgh RT Twitter

Edinburgh RT OSF page

Edinburgh RT mailing list

For any questions/suggestions, please send us an email at edinburgh.reproducibilitea@ed.ac.uk

ReproducibiliTea Blog

Bayesian data analysis and preregistration 17/12/2021 with Dr Zachary Horne

This session was the final session of the year 2021. The speaker was Dr Zachary Horne, a lecturer at School of Philosophy, Psychology & Language Sciences, The University of Edinburgh. Dr Horne talked about Bayesian statistics and preregistration in the context of open research practices. Dr Horne started his presentation by talking about what is Bayesian data analysis and very broadly it is a data analysis that takes into consideration prior information about a particular domain, in addition to data collection. Sometimes it is also called prior distribution. 

There are different aspects to keep in mind when it comes to preregistration in Bayesian data analysis:

  • How the data is going to be collected
  • Why is the data being collected in a particular way?
  • Sample size
  • Operationalization of constructs
  • Specifying key analyses
  • Aspects of analysis that will be exploratory

Bayesian workflow (Gelman et al., 2020)

  1. Choosing an initial model
  2. Prior predictive checking
  3. Fitting the model
  4. Computational problems and algorithm diagnostics
  5. Posterior predictive checking
  6. Prior robustness

Dr Horne talked about prior predictive checking in a bit detail and it covers the following features:

  • Prior to data collection, is the model consistent with what is already known about the world?
  • What distribution is implied for an outcome variable given prior and likelihood?
  • Assessing the credibility of model before collecting the data

A question ‘Do tweets from activist groups (e.g., PETA, Greenpeace, etc.) with photos get liked more than tweets without photos?’ was central during the session to discuss models in Bayesian data analysis. Analysis showed that photos are better as far as likes are concerned on twitter. With respect to which model is the ‘right’ model, the regularizing model provided better estimates of central tendency of distribution. However, none of the priors (optimistic, regularizing and improper) captured that the larger central tendency is coming out just from many tweets getting 200 or so likes, but also from tweets getting huge numbers of likes! Moreover, models have a room for improvement. 

The session was concluded with pre-registration priors in Bayesian data analysis and Dr Horne suggested using regularizing priors for the parameters of interest especially when those parameters are expected to ‘do something’ and incorporating posterior information in the priors of subsequent related models.

This blog is written by Sumbul Syed

SOCIALS:

Edinburgh RT Twitter

Edinburgh RT OSF page

Edinburgh RT mailing list

For any questions/suggestions, please send us an email at edinburgh.reproducibilitea@ed.ac.uk

ReproducibiliTea Blog

Edinburgh University Research Optimisation Course (EUROC) 19/11/2021 with Dr Gillian Currie

In this session, Dr Gillian Currie who is a Postdoctoral Research Fellow in the CAMARADES group, Centre for Clinical Brain Sciences at The University of Edinburgh talked about EUROC (Edinburgh University Research Optimisation Course) which encourages open research practices in animal research. Dr Gillian Currie is a meta-researcher and her research interests include improvement in research methodology.

Dr Currie began talking about EUROC (Edinburgh University Research Optimization Course), a course with a focus on rigorous design, conduct, analysis and reporting of research using animals. She further mentioned some key points on research using animals:

  • In the year 2020, 2.8 million animals were used in research across the UK
  • The studies helped understand basic biology, complex diseases and potential treatments development
  • However, there were certain concerns regarding difficulties in replication, reproducibility and translation

Dr Currie talked briefly about the translational pipeline which aims to translate pre-clinical research into clinical research which further results in improved health. A survey conducted by Nature involving 1,576 researchers found that 52% of the researchers agreed there is a ‘reproducibility crisis’. The problem of ‘replication crisis’ can be attributed to the following reasons:

  1. Smaller sample size in studies
  2. Publication bias
  3. Limited randomization and blinding

Dr Currie carried on with a discussion by talking about new opportunities in open research practices including:

  1. An increased focus on methodological rigour which involves ensuring appropriate power, appropriate statistics and p values
  2. An increased transparency through pre-registration of studies, reporting of methods as well as sharing of data
  3. Measures to reduce risks of biases

It is important to realise that a small improvement manifested across large number of researchers can help make sure to have a substantial effect overall. 

Course structure of EUROC:

EUROC comprises of 3 modules which can be completed across multiple sessions. Every module consists of 1 core and 1 extended lecture.

MODULE 1: Study Design and Data Analysis

In module 1, ‘Study Design’ section will comprise of internal validity, Risks of bias, Construct and external validity and Exploratory vs confirmatory research. ‘Data Analysis’ section consists of Statistical analysis, Significance testing, Sample size and statistical power, Outliers, Unit of analysis and Multiple outcome testing.

MODULE 2: Experimental Procedure

Module 2 is divided into two sections: Maximizing Study Validity and Study Design. The former section includes topics like Risks of bias, Pilot studies, Confounding characteristics and variables, Validity of outcome and Optimization of complex treatment parameters. The latter section will have Use of reference compounds, Statistical Analysis Tips, Replication and Standardisation.

MODULE 3: Pre-registration and Reporting

The final module will deal with Pre-registration (including Study protocols) and Reporting (Data sharing, Statement of conflict of interest, Reporting standards).

The course is a contribution by The University of Edinburgh towards an improvement in research. Therefore, the course is also available to researchers outside the university through this link edin.ac/2SZvY4U.

How to access EUROC on Learn (for people within the University of Edinburgh):

  1. Log in to Learn
  2. Click on ‘self-enrol’ (available on top right of the screen)
  3. Scroll down to Research Improvement
  4. Click on EUROC (Edinburgh University Research Optimisation Course)

The session was concluded with Dr Currie talking about a research improvement project that is coming up soon. Delays in dissemination of research findings act as impediments in scientific progress, therefore one of the most important aims of research improvement project is to increase the speed at which findings are shared with the use of pre-prints. A Pre-print is an early version of a scholarly article that has not gone under peer-review. It is open to comments and is a good means to prioritise new ideas.

This blog is written by Sumbul Syed

SOCIALS:

Session’ video on YouTube

Edinburgh RT Twitter

Edinburgh RT OSF page

Edinburgh RT mailing list

For any questions/suggestions, please send us an email at edinburgh.reproducibilitea@ed.ac.uk

ReproducibiliTea Blog

Building an Open Research Culture 29/10/2021 with Dr Will Cawthorn

In this session, Dr Will Cawthorn at The University of Edinburgh Centre for Cardiovascular Science talked about Building an Open Research Culture. Dr Cawthorn began by talking about the conflicts of interest in research which can be extrinsic and intrinsic and how important it is in open research to have no potential conflicts of interest. Here are the key points which were discussed during the session:

  • Value of a research study these days is based a lot more on if it’s published in a high impact journal and whether it is highly cited
  • Experts opinions are usually flawed especially in flawed and noisy environments
  • There are many consequences of mismeasurement of science including devaluation and ignoring of valuable research, publication delays, incentivization of poor research practice and external pressures killing inner motivation to do good research 
  • Researchers pay to publish their research they produced in the first place in a journal only to ask others to pay in order to access it. This is the opposite of open research

Dr Cawthorn further talked about how he is taking the steps to bring about an open research culture in his own lab. 

  • Encouraging members of his lab to follow their own ideas
  • Publishing negative results because there is no such thing as ‘positive’ and ‘negative’ results. Just the ‘conclusive’ and ‘inconclusive’ results.
  • Encouraging more preprints and open access papers

However, open research culture is easier said than done and there are additional practices that Dr Cawthorn wants to introduce/further improve in his lab. And these include:

  • Electronic lab notebook
  • Writing a lab manual
  • Robust data management

The session was concluded with a question regarding if there is any brighter future of open research culture. There is probably a brighter future with many initiatives coming up like DORA (The Declaration on Research Assessment) and LERU (The League of European Research Universities).

Dr Cawthorn is the LERU Open Science Ambassador for The University of Edinburgh and he is collaborating with many others on writing an Open Science Roadmap for the University which is due to be published soon.

This blog is written by Sumbul Syed


SOCIALS-

Session’s video on YouTube

Edinburgh RT Twitter

Edinburgh RT OSF page

Edinburgh RT mailing list

For any questions/suggestions, please send us an email at edinburgh.reproducibilitea@ed.ac.uk