The Challenge
Scientists communicate findings through publications in peer-reviewed journals. However, this process is slow and prone to biases as it relies heavily on existing social networks of scientists. It also relies on the unpaid labor of researchers who are not incentivized to write thorough or well-thought-out reviews, which can lead to inconsistent or unhelpful reviews. Peer review can also be biased by the status, institution, or demographics of the authors. This process hampers scientific progress. To combat some of these issues, scholars have begun to post pre-prints which allows researchers to post a copy of their paper at any stage. However, this can lead to public dissemination of findings that have not been confirmed by other scientists. All of this came to a dramatic and explosive point in the controversies about preprints and pandemic research. We will consider challenges associated with peer review, and tools designed to hold on to the good parts while jettisoning the bad.
Readings
- “Peer review isn’t working.” Everything Hertz Podcast episode 159, with Saloni Dattani.
- A Twitter thread on evidence of status bias in peer review.
- A humorous approach to problems with peer review: Shit My Reviewers Say [Tumblr page].
- Bengio, Y. (2020, February 26). “Time to rethink the publication process in machine learning.” [Blog post]
- By contrast, an example of the spread of false information in non-peer-reviewed sources is the infamous Yan report.
- Read over eLife’s description of their ’new model’: [Link to eLife’s description] and about their assessment model: [Link to eLife’s assessment model].
- Look through Preprints and Rapid Communication of COVID-19 research, a collection of articles and resources by ASAP Bio.
In sum, peer review is slow, prone to error, potentially biased, and there’s no incentive for anyone to do it well, but doing without peer review is also problematic! In your response paper, describe specific examples of when and why peer review and/or rapid dissemination of results are important for your science, something you learned from the readings you didn’t already know, and/or a personal experience you’ve had that really brought home the challenge of peer review and/or rapid dissemination of results.
Somewhat older resources:
- Vale, R. D. (2015). “Accelerating scientific publication in biology. Proceedings of the National Academy of Sciences, 112(44), 13439-13446.
- Kaiser J. (2017). “The preprint dilemma.” Science. 357:1344–1349.
