Tools for Robust Science

Week 3 Tool: FAIR Protocols; Methodology Reporting Standards

The Tool

Practical skills assignment

1. Find a paper in your field that describes an experiment. Use the ARRIVE and CONSORT checklists (PDF) to evaluate how well they communicate their methods and design decisions. What is missing? Note any trouble you have applying these checklists to research in your area.
2. Find the website or tool that is used for sharing detailed/executable protocols in your discipline (see examples below). Ideally, find an existing template similar to an experiment you have done or are planning to do. Spend 30 minutes beginning to create an experimental protocol in this tool. In response paper answer: How far did you get? What was difficult about the task?

In your response paper, describe what you accomplished in these tasks and any snags you hit.

Then, critically evaluate these tools: will they address the obstacles to repeatability in your area of science? What else is needed? 

Useful Links and Resources

• Tools for shared protocols:
  • Biological sciences: Protocols.io, ProtocolExchange, & Protocol-online.
  • Online behavioural experiments: OTree PsyToolkit Pavlovia or PCIbex.
  • Computational biology: nextflow.
  • List of electronic laboratory notebook software packages: Wikipedia page.
• ARRIVE and CONSORT methodology reporting standards: ARRIVE Guidelines  & CONSORT Statement.
• Videos of workshops on how to improve reproducibility.
• Repository of scales used in Psychology.
• Collection of articles about reproducibility.
• https://datajoint.com/
• https://www.nc3rs.org.uk/experimental-design-assistant-eda
• Interview with Teytelman where he talks about the obstacles to the uptake of protocols.
• A slate article about how personal and ugly the controversies about replication can get. This story is about a psychology experiment about moral priming, and the controversy was called “#repligate”. 
• More than twenty years ago there was a famous controversy about the replicability of findings in genetically identical mice. Original paper; Follow-up paper.
• Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015). “The economics of reproducibility in preclinical research.” PLoS biology, 13(6), e1002165.
• Ashley Thomas’ example (in development): Conducting Online Research With Infants.