By Paul Homewood

h/t stewgreen

The Times Higher Education Supplement’s blog carries a damning indictment of peer review, which has very real relevance to the climate change debate:

Given the entirely appropriate degree of respect that science has for data, the ongoing discussion of peer review is often surprisingly data-free and underlain by the implicit assumption that peer review - although in need of improvement - is indispensable.

The thing is, the peer review of scientific reports is not only without documented value in advancing the scientific enterprise but, in a manner that few care to acknowledge openly, primarily serves ends that are less than noble. Peer review is widely assumed to provide an imprimatur of scientific quality (and significance) for a publication, but this is clearly not the case.

While the many flaws of peer review are clearly laid out in the literature, its failure to protect the integrity of the scientific enterprise is notable. An estimated cost of irreproducible biomedical research is $28 billion (20 billion pounds) a year and ‘currently, many published research findings are false or exaggerated, and an estimated 85 per cent of research resources are wasted’, one paper found.

A prime example of the failure of peer review is the tainting of a significant segment of the biomedical literature by the use of misidentified and contaminated cell lines pointing, at best, to a culture of carelessness in cell biology research and the clear failure of peer review to discover and correct erroneous research.

There are many reasons why scientific peer review is ineffective. An important factor is the inadequacy of almost all scientific reporting; publications should contain sufficient information that all aspects of the work can be understood, permitting a published result to be reproduced from the original data, as well as independent replication of the study by others wishing to do this.

If these minimal standards are not met then critical information is missing and the reader has no way of assessing if the published research is correct or false in its claims and conclusions - even exact replication of a study is precluded. Reproducibility and repeatability require that all theory, methods, equipment, reagents, source code, computational environment, raw data and analytical and statistical methods be fully documented and openly available.

This standard is not enforced by peer review as currently practiced, with the result that most publications in most journals should be viewed by the skeptical reader as little better than advertisements that present the authors’ claim to priority but preclude straightforward and independent verification.

We are not the first to identify these problems, so we might ask why peer review retains its essentially unassailable status. We suggest a two-fold answer rooted more in socio-economic factors than the dispassionate review of scientific research.

First, peer review is self-evidently useful in protecting established paradigms and disadvantaging challenges to entrenched scientific authority. Second, peer review, by controlling access to publication in the most prestigious journals helps to maintain the clearly recognized hierarchies of journals, of researchers, and of universities and research institutes. Peer reviewers should be experts in their field and will therefore have allegiances to leaders in their field and to their shared scientific consensus; conversely, there will be a natural hostility to challenges to the consensus, and peer reviewers have substantial power of influence (extending virtually to censorship) over publication in elite (and even not-so-elite) journals.

Publication in the highest-profile journals reinforces the hierarchies of status in the scientific community and promotes very effectively the prestige-, career- and profit-driven motives of authors, journal editors, publishers and (less directly) universities. This state of affairs exerts a particularly baleful influence on interdisciplinary research.

Innovations in peer review (including dispensing with its traditional forms) are to be encouraged. It may be that open publication through servers such as arXiv and bioRxiv, along with public and signed post-publication comment, are the solution to the problems noted above.

However, for any innovations in scientific publication to succeed two conditions would need to be met. The first, as noted above, is the provision with a publication of all the information necessary for independent reproduction and repeatability of the research, and the second is the improvement in the culture of science such that less than rigorous work and deceptive publication practices are no longer tolerated.

With the scientific method itself at risk, the stakes could not be higher.

Les Hatton is a mathematician and emeritus professor at Kingston University. Gregory Warr is a biochemist and emeritus professor at the Medical University of South Carolina. Both have extensive experience as peer reviewers and journal editors.

See more here and here.