The Challenge
A defining feature of scientific knowledge is that it is replicable: another scientist should be able to repeat the procedure and re-generate the same results. Yet in reality, there are many obstacles to replication. The first obstacle is often: published procedures are not described sufficiently, to allow another scientist to reproduce them. Other obstacles to replication will be discussed in later classes (e.g. experimenter degrees of freedom and publication bias mean that published results offer an inflated estimate of effect sizes; professional incentives for scientists often don’t reward reproducing or being reproducible). Today we focus on the challenge of how to make scientific procedures reproducible.
Readings
Everyone should read the resources under ’everyone.’ After that please read the additional readings most relevant to you.
Everyone
- Nosek, B. A., & Errington, T. M. (2020). “The best time to argue about what a replication means? Before you do it.” Nature 583, 518-520.
- Teytelman, L. (2018). “No more excuses for non-reproducible methods.” Nature 560, 411. Alternatively, watch a recent (2021) talk (the beginning is funny/sad).
Biological Sciences
Errington, T. M. (2019). “Reproducibility Project: Cancer Biology, Barriers to Replicability in the Process of Research.” Talk at the MetaScience Conference. Read the slides or watch the talk.
Systems Neuroscience
Banga, K., Benson, J., Bonacchi, N., et al. (2022). “Reproducibility of in-vivo electrophysiological measurements in mice.” bioRxiv.
Computational Science
- Barton, CM., et al.(2022). “How to make models more useful.” PNAS 119(35) e2202112119.
- Pineau, J. (2020). “The Machine Learning Reproducibility Checklist.”
Developmental Science
- Gilmore, R. O., & Adolph, K. E. (2017). “Video can make behavioural science more reproducible.” Nature human behaviour, 1(7), 1-2.
- Frank, M. C., Bergelson, E., Bergmann, C., et al. (2017). “A collaborative approach to infant research: Promoting reproducibility, best practices, and theory‐building.” Infancy, 22(4), 421-435. Alternatively, watch this talk by Mike Frank.
Social Science
Moody, J. W., Keister, L. A., & Ramos, M. C. (2022). “Reproducibility in the Social Sciences.” Annual Review of Sociology, 48.
