Seen elsewhere

 

Buy

Books
Click images for more details

Twitter
Support

 

Recent posts
Recent comments
Currently discussing
Links

A few sites I've stumbled across recently....

Powered by Squarespace
« Hosing down beancounters | Main | Ed Davey on Sunday Politics »
Monday
Jul152013

Improving peer review

Discover Magazine considers a new proposal for eliminating some of the bias from the peer review process.

...scientific manuscripts should be submitted for peer review with the results and discussion omitted. The reviewers would judge the submission on the strength of the methods and the introduction alone. If they recommended publication, the authors would then send them the full paper.

The reviewers would then have a chance to change their mind and reject it, or ask for further experiments to be carried out, but the ‘bar’ for this to happen would be high.

This is quite a neat idea, potentially removing much of the bias in the peer review process. Other problems would, however, remain. For example, You can imagine a paper that began "we updated all the major tree ring temperature proxies in North America and northern Europe" being rejected out of hand - there are some questions that people just don't want answered.

PrintView Printer Friendly Version

References (1)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    - Bishop Hill blog - Improving peer review

Reader Comments (33)

It would require a half-way house to cleanse any information about the submitter so the reviewers have no clue regarding the source. It would not do to have a submission in the following way:

"Hey, guys Mike here. Give this a glance and shoot me your comments and lets get this rammed through asap. As always, a pleasure working with you. Kevin - good to see you last week at Cancun - next time bring some sun screen - OUCH! :)"

I've made up the names to protect the innocent.

Jul 15, 2013 at 8:33 AM | Unregistered Commenterdp

Well, no. The most important part of a study is indeed the methodology. However, removing all else from a manuscript the paper does absolutely nothing to resolve the core problem, that is, the peer-review system itself. What we need is accountability and that requires a transparent procedure. Eliminate the false "blinds", I say.

Jul 15, 2013 at 9:29 AM | Unregistered CommenterBrute

"Discover Magazine considers a new proposal..."

Not exactly a NEW proposal - in the text it says:

...It’s a clever notion (and, as Smulders points out, not a new one; it dates to the 1970s, but has never taken off.)....

Jul 15, 2013 at 10:21 AM | Unregistered CommenterDodgy Geezer

There's somehow been a misconception about "peer review" and that is somehow means that the paper is correct because it has been "reviewed". Peer reviewing is not the same as an engineering review, in that the content of the paper does not need to be correct. This may sound strange to anyone who hasn't published but the whole idea of peer review and publishing is that you put forward your work and as long as it's not glaring bad it gets added to the body of scientific information.

Whether it is wrong or right is taken on by people repeating the work or auditing it (a bit like what Steve Mc does). It's only in "climate science" that people have used peer review as being a measure of the paper being correct. And actually all this demonstrates is that people who do, don't understand the progress of science or have become advocates for a certain way of thinking and are just tallying up a "score" of papers that endorse their opinion.

Another thing: it is better to have papers out there that are dead wrong. Sometimes they get so many references because they demonstrate what NOT to do. The recent Marcott paper should stay as a shining example of good intentions but bad methodology and conclusions. You can then tell the calibre of people who reference it blindly.

Jul 15, 2013 at 10:31 AM | Unregistered CommenterMicky H Corbett

Micky H Corbett,

Yes, it was just a way by which journals ensured they were publishing something new, interesting and not obviously tripe. Lots of peer reviewed papers were found not to be replicable or the experimental methods were shown to be invalid.

I think this idea of peer review as a stamp of undisputable correctness came from the arts. It feeds the idea that science is done by consensus of the wise and the good.

Jul 15, 2013 at 10:43 AM | Unregistered Commentercosmic

Agree with Micky H Corbett - In fact I would go one stage further in that because the peer review process became demonstrably "Pal Review" in Climatescience, that is why many started to question the "science"

How I cringed at the Alarmists twin mantra of "Peer Review = Consensus" which lead to the unbelievable "The Scientists have spoken!".

There are enough people in the hard scientific disciplines now looking at the soft science that is Climatescience for us to be reasonably confident that the truth will emerge.

Jul 15, 2013 at 10:47 AM | Unregistered CommenterDoug UK

No way, because the hidden results may no correspond to the described method. Far too much work for the reviewer, first he has to mentally co-invent the method along with the authors, then envision how the results should go, then, when he's already "reviewed" it, evaluate the actual results and fit them to his own preconceptions. Sorry, it's hard enough to get reviewers.

Jul 15, 2013 at 10:56 AM | Unregistered CommenterNZ Willy

A good peer review would be independent experiments supporting a paper. Else. a warmomg, in font several sizes larger than the headline. stating "I'm Guessing. Maybe even SWAG."

Jul 15, 2013 at 11:18 AM | Unregistered Commentercedarhill

I suggest that papers should only be eligible for peer review, once the data, codes etc are availvable, to ensure that interested parties have sufficient information to replicate the paper.
This procedure would ensure that the authors paid more attention to their work.

Jul 15, 2013 at 11:35 AM | Unregistered Commenterpesadia

I suggest that papers should only be eligible for peer review, once the data, codes etc are availvable...

A bit like a mining prospectus, no?

Доверяй, но проверяй, comerade academician.

Jul 15, 2013 at 12:11 PM | Unregistered CommenterManniakov

Some kind of review is necessary for any publication, otherwise complete rubbish from pranksters and mono-maniacs will swamp any useful stuff.

Most publications - magazines and newspapers, for instance, have their own reviewers on staff who know about their own specialist interest. Try getting a claim that you surfed a 200 ft high wave off Cornwall into a surfing magazine, for instance, and the editor will soon start asking questions. You won't get very far claiming that you have 10 1856 British Guiana 1 cent Magenta stamps in your collection to a philately magazine. It ought to be the same with ALL technical subjects.

The problem is that most of the journals we are concerned about are not run by people who love their subject - they are run as money-making concerns. Staff on ANY specialist journal should easily be able to identify suspect papers. Occasionally they might need to refer a piece to an external specialist - but they ought to know who the best specialists are for each issue, and be able to take responsibility for publishing controversial papers - that's why they are running that journal and maintaining it's reputation.

Newspapers are quite happy to work that way - they check out news and publish - and take the reputation hit if they are found to be publishing propaganda as fact. What is so special about Science journals which lets them charge huge cover prices, and then absolves them from responsibility for anything they publish?

Jul 15, 2013 at 12:15 PM | Unregistered CommenterDodgy Geezer

Jul 15, 2013 at 10:31 AM Micky H Corbett

Agreed 100%. In past existences I have published plenty of papers which were refereed prior to publication and I have reviewed lots of papers for journals.

I had never heard the term "peer reviewed" until I started to clue myself up on climate science some years back. It seems to have originated with the IPCC boasting that they only refer to 'peer-reviewed' publications. The climate science community seems to have adopted it in the sense of:

"peer reviewed" = "validated and certified correct".

As someone on BH suggested recently, it might well become a pejorative term, in view of its association with "climate science". (The latter does not really, in my view, justify the term "science" in view of its reliance on the output of unvalidated models as "evidence" and its general avoidance of the scientific method.)

Jul 15, 2013 at 12:16 PM | Registered CommenterMartin A

I'm 100% with Micky H Corbett here.
And I know of what I speak.

Jul 15, 2013 at 12:19 PM | Unregistered CommenterLevelGaze

As others note above, a significant problem is how peer review seems to viewed by those less familiar with it. It's main value is a filter which helps save other scientists time, by not having to read so much that would really just be wasted effort. It's not a gold standard, it's a minimum standard. You still have to do your own thinking.

Irrespective, however well or badly peer review works, that won't stop exaggerated claims being made elsewhere.

Jul 15, 2013 at 12:26 PM | Unregistered Commentermichael hart

How will that work? The reviewers want to check not just the method, but what conclusions (right or wrong) were arrived at from there, and the spin the authors put on it.

Jul 15, 2013 at 12:36 PM | Registered Commentershub

Peer review in climate science requires that the author (s) accept the 'Consensus'.

It's a bit like the Concordat in 17 th Century Rome insisting Galileo support the heliocentric view of the universe.

Jul 15, 2013 at 12:51 PM | Unregistered CommenterAlecM

The implication is that "peer review" has some cachet, and we should be supporting and strengthening it, even though it says nothing about the validity of a paper.

I have no objection to peer review, but using it as a proxy for authenticity is a bridge too far. Let's maintain that distinction.

Jul 15, 2013 at 2:02 PM | Unregistered Commenterjohanna

It is clear that the whole system needs tearing up and replacing with something not designed by Heath Robinson.

OK my name is Heath Robinson and these are my suggestions:

Several levels of review each one clearly defined and explained

Lowest: Peer Review. This paper has been "looked at" by a competent scientist to whom the conclusions seem reasonable and who found no obvious fault with the methodology.

Middle: Specific Review. Each aspect covered by this paper has been looked at by a scientist qualified in that field and who found the conclusions reasonable and found no obvious fault with the methodology. (All data and software used to be supplied)

Gold Standard: (payment of fees required) Fully analysed, Each aspect of this paper has been analysed by a competent scientist qualified in that field.
Data verified
Data selection verified
statistical/analytical methods verified
All results replicated
No fault can be found with the conclusions.

Jul 15, 2013 at 2:58 PM | Registered CommenterDung

Relating to my Fully Analysed Review above, there would need to be some kind of accreditation system for reviewers and an appeal system both for authors and critics. a successful appeal showing a reviewer got it wrong would result in the reviewer losing his/her accreditation and the ability to earn fees.

Jul 15, 2013 at 3:09 PM | Registered CommenterDung

+100 for everyone who has said similar.

Peer review is a workflow process in the academic publishing industry. Bit like stapling or loading in a van.

Anyone who thinks that publishing or peer-reviewing a claim makes something true has zero understanding of science.

Jul 15, 2013 at 3:30 PM | Unregistered CommenterJack Hughes

Peer Review in most of science works just fine and does what it was originally intended to do; is this work novel, have the authors looked at other work in this area, have the provided sufficient detail of the methods for the work to be reproduced by others, are there any failures in logic in moving from results into discussion.

It "works" because the vast majority of published papers in science add very little value to the scientific canon, and in the majority of cases the work is of little relevance and rarely cited by others.

However, where leading edge scientific work is going to be immediately used for regulation or to support vast amounts of investment we need to have a more robust system that demands a higher level of proof. Climate scientists, NGOs and politicians still don't understand that data and conclusions need engineering standard review rather than conventional peer (pal) review if the public is going to tke the conclusions seriously.

Jul 15, 2013 at 3:30 PM | Unregistered CommenterArthur Dent

Arthur Dent

The "engineering standard review" is actually a great analogy. If you consider the current spending on "climate mitigation" as a project spend to "solve an issue", then in any engineering review the first thing that is looked at are the requirements. And if they are valid.

Validity would come from supporting information and test data. In space projects this is called Pre Phase A data - the stuff that comes with a bid or at the start of a project when requirements are a bit more fluid.

The attempts to reduce CO2 warming would first of all need a good body of evidence that CO2 warms before money would be released. If you were to do this now, would you release the cash? I know I wouldn't with the current available data.

Jul 15, 2013 at 4:20 PM | Unregistered CommenterMicky H Corbett

I agree with the comments above regarding efficacy of "peer review."

To apply an Americanism to this proposal, I think it is nuts. If one assumed the referees to be truly arms-length (couldn't identify the authors from the content), adopting this method would be to hold back information which might be required to determine the level of care needed for the review. Time would be wasted.

If I understand this genre of literature correctly, conclusions unsupported (unsupportable) by the research is common. Wouldn't one think that this form of error would be more readily detected by exposing the entire paper to review?

Jul 15, 2013 at 5:03 PM | Registered Commenterjferguson

Yes, and Steve McIntyre's recent posts on the latest Briffa (ha) paper and the need to vastly improve the quality, quantity and transparency of tree ring data is, one must conclude, largely a work in the rhetorical style.

It is simply not in the interests of climate activists and their pseudoscience lackys to have any stage of the process be other than opaque, biased and politicised.

Many a career and pension entitlement depends on this state of affairs.

Jul 15, 2013 at 5:24 PM | Unregistered CommenterPhill and Keith's "Career Trick"

There is an incipient, general revolution in science, a NEW PARADIGM (yet very old; I call it "The Once and Future Paradigm"), that needs to come out, and peer-review -- any review by current deluded and dogmatically defensive "experts" -- is incompetent to handle it. Imagine back in Galileo's day, "peer review" was by ecclesiastic overseers; the situation is similar to that. We are dealing with an incompetent fundamental belief system, maintaining an incompetent consensus, across many fields of science, and climate science is just the tip of the iceberg, with its deluded assumption of a globe poised on the razor edge of "runaway global climate", either too hot OR too cold. No isolated debate, over any one supposed "cause" or "effect" (such as the "greenhouse effect" debate in climate science) can properly address the underlying general incompetence (that makes both alarmists and skeptics fellow-travellers in the "global-warming greenhouse effect" delusion). Adding epicycles to peer review is meaningless in that larger, proper context.

Jul 15, 2013 at 5:53 PM | Unregistered CommenterHarry Dale Huffman

OK Harry I will just get my coat if that's OK?

Jul 15, 2013 at 8:15 PM | Registered CommenterDung

You would need to have authors cloak themselves with noms de plume. Otherwise picture what would happen when Dick Lindzen submitted a paper to GRL.

No gimmick will work. The system needs to collapse first, which is where it is headed.

Jul 16, 2013 at 1:04 AM | Unregistered CommenterNoblesse Oblige

While I agree with Micky Corbett I think a simple solution to the issue is to have complete transparency in the process. The reviewers are named , their comments are published ( even if only in the online version of the paper) , the authors response to the reviewer is published and any editor comment is also published. This process is followed by a couple of journals that I know of -- I am not an author of published papers.

Jul 16, 2013 at 2:18 AM | Unregistered CommenterRoss

"Pay per view" ... reviewers should receive a stipend for undertaking a review, they should be named, and they should be accountable for shoddy outcomes. This will help replace some of the academic grant revenues that will evaporate when 'climate science (TM)' implodes.

Jul 16, 2013 at 2:25 AM | Unregistered CommenterStreetcred

Peer review = good/correct was a deliberate tactic by Schmidt et al, all part of the fooling of the public who could not know any better, to discredit any criticism of their work/policy agenda. In particular it was (as far as I remember) first wheeled out as a smack-down of the M&M paper published in E&E, the warmists wholly dishonestly asserting that E&E was not peer-reviewed, and using as evidence for that the then correct fact E&E was not in ISI, all journals in ISI are peer-reviewed, E&E not in ISI therefore not peer-reviewed. As 'Professor' Jones so succinctly put it, in a slightly different context "I'll re-define (what constitutes) the peer-reviewed literature if I have to", or words to that effect. In short, at every level, the warmist camp is shot through with intellectual dishonesty.

Jul 16, 2013 at 3:02 PM | Unregistered Commenterbill

You say "The reviewers would judge the submission on the strength of the methods and the introduction alone. " I would continue this as follows:

Once approved, the work would be contracted out to a paid, independent third party; once completed, it WILL be published, no matter how it comes out. As far as the issue of how to pay for this:
Since no more work is being done than before, it shouldn't cost more than before. For example, maybe my graduate students will earn money by doing someone else's experiments, just as I will (in effect) pay someone else's graduate experiments to do mine. In fact, I would guess there will be less work done than before, since some "results" that were really just the result of absurd data mining or of experimental fraud would never have been proposed in the first place.

Jul 16, 2013 at 8:49 PM | Unregistered CommenterLTEC

Sorry. One of the most important parts of peer review is deciding whether the conclusions drawn in a paper (especially those in the abstract) are justified by the data disclosed in a paper.

Peer review also needs to consider whether the appropriate caveats have been included and the results has been properly considered in light of the existing literature. Papers that produce unexpected or ground-breaking results are extremely attractive to prestigious journals.

If might be interesting if one reviewer were given only the introduction and methods. (The introduction and methods are what makes up much of a grant proposal, so that reviewer is essentially being asked whether the grant should be funded.)

In human clinical trials with new drugs, researchers are now required to prepare a site in a public depository for all the data they will collect, the methods they will use to analyze the data, and the projected date data will be deposited.

Jul 17, 2013 at 12:07 AM | Unregistered CommenterFrank

The whole "gentlements agreement" ethos of peer review is an anachronism. I understand how we got here, but If you want to clean up peer review:

1. Stipulate in the funding agreement for all government funded science that everything required to replicate the results be uploaded to a central on-line respository *prior* to press-release or publication.
2. The publishing instutution pays a repository a publication fee of £1,000 (to cover hosting and expenses) and a deposit of, say, £10,000
3. If the paper remains un-debunked after 12 months the publishing institutions deposit is returned. If it's debunked as garbage on WUWT within 24 hours the publishing institution loses their desposit (which is paid to the debunker) and the institution forfeits the right to publish to the repository or to receive further new government contracts for 6 months.

This would create a space in the market for business to be created employing scientists / statisticians / engineers etc as professional debunkers, profiting from the deposits from factually challenged authors.

This would also give the home institutions of factually challenged authors cause to look closely at the the scientific value of their staffs output.

Jul 18, 2013 at 12:30 AM | Unregistered CommenterSimon Barnett

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>