written by Laura Klinkhamer (co-organiser of Edinburgh ReproducibiliTea)
In this session we took a look at the following paper:
Ummul-Kiram Kathawalla, Priya Silverstein, Moin Syed; Easing Into Open Science: A Guide for Graduate Students and Their Advisors. Collabra: Psychology 4 January 2021; 7 (1): 18684. doi: https://doi.org/10.1525/collabra.18684
and we were joined by one of the authors, Dr Priya Silverstein for a live Q&A.
The paper is a great place to start for people who are new to open research concepts and provides a very useful summary and guide for some practices you could consider applying to your research. It introduces open science (now often referred to as open research, to include the academic disciplines that would not describe themselves as a “science”), as a “broad term that refers to a variety of principles and behaviors pertaining to transparency, credibility, reproducibility, and accessibility” (Kathawalla et al., 2021, p. 2). The paper is written specifically for the types of situations that graduate students are more likely to encounter, but the practices described are broadly applicable to researchers of any career stage.
Paper Summary
Eight Open Research (OR) practices are outlined in this guide and classified according to the author’s perception of the difficulty of implementation.
Practice 1 is to set up or join an open research (journal) club, such as the with the ReproducibiliTea organisation. This can be a quick and efficient way of getting to grips with some key concepts of the reproducibility and OR movement, while meeting new people along the way and increasing your network.
For researchers in Edinburgh – we encourage you to join the Edinburgh Open Research Initiative Teams group, which serves as a hub for bringing people interested in OR together. The University of Edinburgh now also has an Open Research Blog and newsletter that you can sign up for here.
Practice 2 refers to thinking about your project workflow, in particular setting up your file organisation, data access regulations and keeping clear records so that (future) you and others can quickly get an overview of your project and are able to reproduce the outcomes. For more information and tips on how to work reproducibly, we refer you to Kaitlyn Hair’s talk (Edinburgh ReproducibiliTea session Nov 2020) on selfish reasons to work reproducibly and Ralitsa Madsen’s talk on RSpace, a platform that you could consider to set up a project workflow in addition to the freely accessible Open Science Framework.
Practice 3 is about preprints, which refers to the practice of publishing your manuscript before or during peer review. Check with the journal where you intend to publish your work first on what their policy on pre-prints is by either messaging them or checking this source suggested by Priya. Pre-prints are a way to bring your research out to the world, even if publication is delayed or rejected. It also increases the number of times your work will be cited. There are free servers that you can upload your manuscript as a pre-print to, such as bioRxiv for biology.
Practice 4 refers to creating reproducible code/analyses. It is very helpful for your project workflow and reproducibility to write your code/analysis plans in such a way that it is clear beyond a doubt for others and your future self what you did. Annotating your steps and writing README files, basic text files describing for instance what files are in your project space/folder and what role they play in your project (e.g. data_spreadsheet_version3 contains the clean data on x number of participants that is used for Analysis B.), are very useful practices.
Practice 5 Sharing data is very useful to the scientific community and there are many platforms that you could upload your project’s anonymized data set to (e.g. OSF again). However, it is very important to make sure you are legally allowed to share the data. This will depend on your local and wider data regulation guides (e.g. in the EU GDPR applies) as well as what has exactly been put into the consent forms (if applicable to your project of course).
There are also options to upload only part of your data set or set up a system so that others can access more sensitive data. Talk to your supervisors/collaborators and check University research support services to see what would be most suitable in your case (for instance see this resource for the school of PPLS).
Practice 6 Being very open in your manuscript writing. In a way it’s fascinating how the norm in manuscript writing is that the research story gets presented as an almost perfect execution of a plan with a happy ending (i.e. significant results), whereas in reality you often hear researchers struggling with all kinds of issues and ending up with a manuscript that is only very slightly connected to the original research idea. It’s not very realistic, and actually harmful to scientific integrity. So if we allow ourselves to be humans, who make mistakes, and allow others to read about and learn from our mistakes, wouldn’t that make life easier?
Practices 7 & 8 are related.
Pre-registration: a time-stamped, read-only version of your research plan created before you begin data collection/analysis.
Registered report: similar to pre-registration but your research plan undergoes peer review before results are known. Helpful resource on the Centre for Open Science website here.
Both practices are very useful ways that make you sit down and plan your research before executing it. In the case of registered reports, you will also obtain feedback before executing it, which may be much more useful than receiving feedback after the fact in the regular peer-review system. Although you state what you intend to research in a pre-registration or registered report, it is important to realise that you do not sign a binding contract. If it turns out that another method or additional exploratory analysis are interesting to your research question, you are of course able to make changes. It is however your responsibility to transparently report and justify these changes.
As Niamh summarised: these practices do not stifle creativity, but create accountability.
It is important to realise that engaging is in OR practices is not an all-or-nothing approach. It’s much more about adopting a certain critical mindset and taking (small) steps that are suitable for you and your specific project.
Discussion
During the discussion Priya mentioned that if the paper were to get written this year she would probably include the same practices, but elaborate on the increased number of options in which they could be applied. For example, one thing that has changed in recent years is that registered reports have become available for projects with secondary data analysis (rather than it just being available for projects where the data still is to be collected).
Another interesting development is that of Peer Community In Registered Reports, which facilitates scheduled peer review. You indicate beforehand when you intend to hand in Stage 1 of your registered report (Introduction & Methods) and the community tries to arrange reviewers for that particular time frame, meaning that the peer review process can be completed much more rapidly. Priya mentioned that in the past this has been one of the main criticisms of the registered report that it was unclear when researchers could start their research analysis/data collection because of uncertainty regarding the peer review duration. This new facility makes registered reports an even more attractive option.
Will Cawthorn added that Review Commons is another place you can send your manuscript to for general peer review. If the manuscript then passes review, you can choose from a list of journals where to publish your paper. This approach decreases redundancy in peer review (i.e. if rejected from one journal, you don’t go through the roulette wheel of another round of completely new peer review).
Priya confirmed that this procedure is also in place for PCI registered reports. From the website: “Following the completion of peer review, authors of RRs that are positively recommended have the option to publish their articles in the growing list of PCI RR-friendly journals that have committed to accepting PCI RR recommendations without further peer review.”
We also had a group discussion about how we could further promote OR in the University. One of the suggested routes was to include more OR practices in undergraduate and postgraduate course curriculae. Will Cawthorn also referred to an OR roadmap for the University of Edinburgh that he and several others (including many from the Library and Research Support Services) are working on that is due to be published soon. Priya emphasised that is important create momentum both through bottom-up and top-down initiatives at the same time to bring about real change in the research culture. This nicely connected to our next session on Friday 15 October, which will be on how to build an open research culture in your lab/research group, by Dr. Will Cawthorn (LERU Open Science Ambassador for the University of Edinburgh).
Then I said something silly about how we should all jump aboard the Open Research train and Priya kindly replied with a “choo choo!” making me feel slightly less embarrassed.
All things considered, we look back at a successful first session of this academic year!
The presentation slides and meeting recording can be found on our OSF page. The meeting recordings can also be found on our YouTube channel.
If you’d like to stay up to date with Edinburgh ReproducibiliTea, please consider joining our mailing list by filling in this form and/or following us on Twitter.
For any questions/suggestions, please send us an email: edinburgh.reproducibilitea@ed.ac.uk
Edinburgh Open Research Initiative 