Bishop Hill SciTech peer review inquiry
Mar 17, 2011 Climate: other Journals The House of Commons Science and Technology Committee have started to publish the submissions of evidence on their website.
A number of familiar names are there, and I'll try to read these when I get a chance:
- Philip Campbell, the editor of Nature who had to resign from the Russell inquiry after prejudging the findings (not to mention his conflict of interest)
- Richard Horton, the editor of The Lancet, who replaced Campbell and whose advice was ignored where Russell found in convenient to do so
- Michael Kelly, of the Oxburgh panel, whose observations on the indequacies of CRU's work was not reported by Oxburgh
- Nic Lewis, of O'Donnell et al fame
- Prominent sceptics, McLean, de Freitas and Carter
There are also two from UEA and one each from the big learned societies, including the Royal Society.
Bishop Hill
Richard Horton's evidence includes this:
Peer review is a central issue in many scientific controversies and disputes today. Take climate change. In the Times Higher Education , last year, Andrew Montford, author of The Hockey Stick Illusion: Climategate and the Corruption of Science (1), argued that events at the Climatic Research Centre (UK) at the University of East Anglia (CRU) had far-reaching implications for the world of scientific peer review and publishing (2). His charge sheet was sharp and precise: that scientists undermined the peer-review process. Implicit in Montford's argument is that peer review is critical to the process of – and thereby public trust in – science. Writing in The Guardian , George Monbiot put it this way: "science happens to be [a] closed world with one of the most effective forms of self-regulation: the peer review process."(3). But for many of us who do peer review, this "most effective" form of self-regulation is often misunderstood and misrepresented.
Reader Comments (18)
Wot No Hat/tip?
At least the Editor of The Lancet (submission No. 2) knows what the issues are. He cites the HSI.
I was putting together a submission, but I ran out of time and missed the deadline. Rats.
I have only skimmed a few submissions so far, but what I have noticed is the propensity to 'puff and bluster' either to the author as an individual or the organisation represented.
I wonder if the Science and Technology Committee will grasp the Nettle?
Wading on through. The McLean, de Freitas and Carter experience is well expressed and typical of the knee jerk hostility any climate sceptic author publishing in the primary journals must face, the editor calling up the 'team' to 'do' the 'rebuttal' routine, but the trouble as always is the difficulty in explaining the injustice. It invariably requires advanced sceptic nerd detail so to do, and that, unfortunately, just bores rank and file politicians stiff, and appears like an over-egged whinge.
As someone who has participated on both sides of peer-review (in a far less contentious field than climate science), I found Richard Horton's evidence to be interesting and fair. I thought the following particularly apt:
Richard Horton's thesis that peer-review is misunderstood and its benefits poorly understood by the general public conforms well to my observations. Do others agree?
I agree entirely, Pluck. It should be self-evident that the peer review process cannot replicate and hence validate research that may have taken a number of years and cost much money and it was never intended to do, although some do appear to think that somehow peer review guarantees the truth of a paper.
In essence, peer review is intended to establish that a paper is worthy of publication and is sufficiently well written and presented that the reader should be able to follow the arguments in the paper. Once published, the work of replication and scientific debate concerning the contents may begin. It is inevitable that some poor or fabricated work will find its way into print because the research would have had the appearance of credibility and worth, and only in the public arena will other scientists be able to devote the time, energy and funds to scrutinise the work in detail.
I don't think any scientist (or engineer) would consider that a paper is undoubtedly true simply because it appears in a peer reviewed journal. I always start reading a paper from the viewpoint that it is wrong and make the authors convince me otherwise. It is not being disrespectful, it simply applies the scepticism which should be part of any scientists nature, and ensures that the arguments are robust enough to overcome my counter arguments.
My son peer reviewed a paper recently and put in quite a bit of research from which he learnt that large sections of the paper had been plagiarised. Those who had asked him to carry out the peer review were grateful but surprised at the amount of effort he'd put in.
In respect of peer review, the difficulties that the anti-CAGW community have experienced arose because reviewers were necessarily experts, almost certainly with opinions that coincided with the non-trivial consensus that was the third pillar of science described by Sir Paul Nurse in his recent interview (This blog 'Paul Nurse on skeptics again' Mar 9, 2011). Peer review is part of the process by which this consensus is established and strengthened. I doubt any general measure can be proposed to ensure that unconventional views are properly evaluated. Moreover, I doubt this enquiry will achieve much - or, indeed that it should. Governments should keep their noses out of science, and a whole lot more.
Ecclesiastical Uncle
Totally agree with the sentiment. Question though: where would we be without building codes and certifications? Buyers would be completely at the mercy of the sellers. Perhaps a certification from a government as to a journal implementing a quality peer-review process would be of some value. Just as a thought: governments could serve the interests of their citizens and even mankind as a whole in this fashion.
Pluck:
Q: "Where would we be without [government] building codes and certifications?"
A: Pretty much where we are now. Because the private sector would step forward and offer such codes and certifications. And insurance companies and mortgage providers would insist on their application.
Jane Coles:
Thanks for the insights.
I agree that it is far beyond any government to propose rational standards (they lack the knowledge and expertise) and that a degree of enforcement is provided by the private sector. Do you believe that a government role is irrelevant or harmful (granted that it can be)? Do you believe that the private sector would succeed without government legislative, administrative and judicial supports?
Personally, I believe that government standards would fail without private sector support. What about the other way round. Would private sector standards fail without government support? Perhaps some government initiatives and support are not helpful. Which ones are?
Thanks, Jane Coles, for your contribution.
Pluck,
The private sector needs a judiciary to enforce contract law. A major disadvantage of government regulation is that the ostensible goal (e.g., safety) gets extended to include political goals. Thus, for example, if water shortage is the political panic du jour, then building codes get a clause insisting on toilets with tanks that are too small to flush properly (so people end up flushing them twice). This kind of nonsense is less likely to happen when there are competing private sector certification schemes.
Apropos science, the problem is not that governments fail to offer a quality certification scheme for publications, but rather that they provide most of the funding for the science in the first place -- see Terence Kealey for the argument.
Jane Coles:
I read, with enjoyment, the reviews on Terence Kealey's work. As someone who engages in research for government and commercial customers, I appreciate some of the differences as ascribed. Actually, it seems to me, that a government requires more that research is publishable, and a commercial client requires more that the research is useful (read that as profitable). Both can be reasonably long-sighted, if the results seem compellingly significant.
I agree with you, Jane Coles, that the judiciary needs to enforce contract law. I suspect that the other branches of government also have important, though abusable, roles. Much of the research I have performed has been funded through legislative sources and directed by administrative managers. Afterwards, some of my results were recognized by commercial sources and further funded by them. I appreciate all who have supported my work, public and private. I recognize that they have different motivations in providing the funding, but none of it, as far as I know, has been politically motivated in any degree. I believe that all of it, public and private, has been motivated by desires to find better solutions to engineering problems.
I suppose that a government's role becomes more contriversial when regulatory measures are being generated. When the measures are enacted without private sector guidance, or inspite of it, a government is certainly capable of doing more harm than good.
By the way, I installed one of those "micro-flush" toilets in my house two years ago. I had to. One of my family members cracked a flush tank. I could not find a replacement part because of government regulations that had banned flush tanks of that size, so I had to replace the whole unit. The new toilet was easy to install and, to my surprise, worked quite well. It uses less than one-fourth of the water. I considered replacing the other three toilets in the house, but didn't. I was too lazy.
Jane Coles:
A further thought on your last comment.
Should I understand your comment to be that governments should not support science funding? If so, though I have benefited from such funding, I am not necessarily opposed to your (inferred by me) condemnation. I am interested, though, given your objections to government funding, in a better way to proceed.
Pluck,
"Should I understand your comment to be that governments should not support science funding?"
Essentially, yes. Babbage, Darwin, Faraday, Gray, Hooker, Kelvin, Lister, Maunder, Maxwell, Rayleigh, Talbot, et al. somehow managed to muddle through without subventions from BBSRC, EPSRC, ERC, ESF, MRC, NERC and STFC. Government funding, in science as in other areas, drives out private support -- see Part III of David Green's Reinventing Civil Society [pdf] for discussion of an important non-science example.
Thank you very much Jane Coles for an interesting and informative dialogue. I have considered your references and enjoyed them. I think that you are quite right: when the government steps in, they tend to make a hash of things. Having enjoyed the support of both private and public sponsors, it would be ungrateful of me not to acknowledge or to thank them for their support.
I like your list of scientists. However, you omitted some of my favorites: John Dalton, Isaac Newton, René Descartes and, of course, Hipparchus. All of them, in their own way, were re-inventors of science.
Pluck,
"You omitted some of my favorites"
My apologies. My list was intendedly C19, that being the closest, temporally and culturally, to C20 and thus the best comparator. But that constraint doesn't excuse the relegation of Dalton to the et al. team!