Bishop Hill Replication, schmeplication
Sep 8, 2013 Journals The Scientist reports on the failure of many scientific papers to include enough information to allow others to replicate the results.
Reproducibility is a hallmark of good science. However, despite the fact that most scientific journals require authors to list the resources used in their experiments, almost half of the papers examined in a new study failed to specify all of the items needed to replicate the findings. The study was published Thursday (September 5) in the journal PeerJ.
From the looks of it, the papers studied were mainly in the life sciences, but this is clearly just as much an issue for climatology. Unless you are an scientivist, of course, in which case replication means being able to reach vaguely similar results using different, but equally obscure methods.
Reader Comments (35)
Yeah, all the way from Life Sciences to Climatology, schmeplication is the future.
There are only two reasons for not giving full disclosure,
1) You forgot to compile a complete list. Unlikely.
2) You knew that you results were not up to standard and tried to hide this salient fact. Probable.
You either do good science or you fudge results to follow your preconceived ideas.
Actually, the key word here is not data but METHODOLOGY.
I recall when the "Journal of Irreproducible Results" was a parody. Now it's gone mainstream.
Detailed instructions not given because of strict space limitations? How 1960s.
Perhaps we should invent something like the Internet that can carry a lot more words in a lot smaller space.
And I don't remember even my old O level science lab books having space limitations. 'Apparatus and Methods' was compulsory, and if I ran out of space in one book, nice Mr Eltringham (Chemistry) would give me another.
There is a story, possibly apocryphal, that I. I. Rabi said that "Science was being able to describe an experiment in a letter to a colleague who could then duplicate the experiment and get the same results."
Surely, replicability is what differentiates science from fiction.
Replication schmeplication
Awesome post title (schmitle)!
When told her son was suffering from a fixation; my elderly neighbour replied: "Oedipus? Schmoedipus! Who - cares as long as he loves his mother."
Just wondering, does BH get the 'word', 'schmord' formation from a misspent youth reading MAD magazine?
A recipe for man made warming.
First there are only a few ingredients but the makers cannot find out what they are and then there are certainly no instructions - because, the method has been thrown away.
"that's the way the money goes"...............
If he did, rhoda (see above) then his youth was most definitely not misspent. In fact, I wonder how many sceptical natures were nurtured by Mad's irreverence?
When the public is paying for science, it would seem only prudent to make replication of results a contractual obligation of research grants. If scientists don't like that they should seek private funding. Either way this is would sort the sheep from the goats, to the benefit of humanity.
The problem with climate science is that many of the most important experiments can't even be done once. "We ran the model twice and got the same results" is probably the closest it gets. Which is why the predictions (what predictions?) are so important.
Fringe times, what a comedy it is these days [no link].
In today's Sunday Times, reporting on Davey trawling round the orient trying to drum up interest in China to build new fission reactors, Russia - Rosatom [God help us] are being sounded too. CHINA, RUSSIA - ENERGY SECURITY?!
Davey, is also reported as saying, shale gas will not be the panacea everyone thinks and we'll have to wait 10 years for a balloon full in any case, the rate we are going - a century hence might look optimistic - what's the problem Ed?
Um....er... could it be the DECC is the problem stalling shale gas exploitation?
Vicky Pope [Met Office], advocating new super computer costing £50 mil' [or £100 mil depending on where you read it] - that will make us all much more secure and computer modelling much more accurate - wasn't that argument they used 4 years ago when they installed the most recent new super-duper @ £30 mill' cost computer??
How much more, how many £millions of taxpayers largesse are we going to be forced to pour down the drain to keep this lot [ the Met Office] in Mercedes and fabulous pension pots?
And there was something else............................... but by that time I was losing the will to live.
When you're a calamitologist, reproducibility might just get in the way of the key message.
If the study had focused on climate science papers, it would have been considered a political work not a scientific one.
Politics is emotions with philosophical excuses.
Um.
This item may be over-egging the pudding a bit. From the article:
...While suggestive of a trend, the study did not, however, report on how often a lack of specific information about experiment resources hindered reproducibility....
Much of the discussion was about how much ancillary data was appropriate to put in a journal which has limited space. The papers discussed were not Climate ones, and there was no indication that the authors would not have provided full details for any aspect of their work if asked. The point made was that not absolutely EVERYTHING needed for replication was included in a published paper, and a reasonable response was: "Is a published paper the best place to put reams of data about reagents?"
So this article does not really support an assertion that the life sciences are going the way of climatology. If anyone thinks of making that assertion...
I have a vision of Maureen Lipman quoting the title with great relish...followed by 'You need an ology, Marvin'
Dodgy Geezer asks the rhetorical question:
'"Is a published paper the best place to put reams of data about reagents?"
Perhaps, in addition to inventing the internet to overcome space limitations, we could invent 'The Appendix' (I realise 'professional scientists' will find such a concept beyond them) and put the relevant stuff in there.
Those who are still finding difficulty with grasping these paradigm shifts will now need to grapple with a third. It is something climatologists should borrow from The Law.
'The Truth, The Whole Truth and Nothing But The Truth'
Now go and have nice cup of Ovaltine, a lie down in a darkened room and wait for your head to stop exploding......everything you ever learnt about the correct way to publish a paper (short, woolly and superficial) is wrong.
Next week: Instalment 2. 'The Junior Climo's Guide to Quality Control' sub: 'Beyond Pal Review...Into the Big Boys Stuff'
Reviewers often ask me to cut out much of my methods sections, even in my methods papers.
I had a reviewer ask me to remove the justification for my choice of statistical testing.
What he hell can you do when the journals want papers a short as possible?
Remember the words ''why should I supply the data , you only want to find something wrong with it '
The rejection of the idea of critical review , that corner stone of science, for 'pal review ' is such a hallmark of 'climate science ' , I wonder just how bad a students work would have to be to get failed . For if the professionals in this area cannot meant the standards expected of an student handing in essay in any other area of science , just how awful must be the students work ?
@Doc Martyn
'What he hell can you do when the journals want papers a short as possible?'
The journals are that way because 'the scientific community' (the consumers of the papers) wants it that way. Not for any other reason.
That is you and your colleagues. The answer lies in your hands. Motivate your coworkers to do it right.
Because - as others have eloquently pointed out - the drift towards pal reviewed puff pieces has seriously damaged the credibility of all scientists in many people's eyes.
Way back when I had to produce all my results, code, methods and data - and be orally examined on them in front of a panel - to get past the examiners. Now Phil Jones reads stuff on the train and passes it 'if it feels right'.
What a pathetic bunch of third raters they are.
OT, but I always have had severe issue with the use of the phrasing such as "Replication, schmeplication". The original, and in my mind only valid, use was:
"-ism? Schism!"
This is very clever, as it very neatly defines any '-ism' as resulting in a schism. I have always felt that any bastardisation of that jewel is at best a minor insult to the original creator (Could be Bob Marley?), but I suspect it's just me.
Climate Porn is becoming more and more like Internet Porn ,more extreme and more people getting bored of it.
Being tight for space is a pathetic argument. Presumably if this is the case then a link to the data could be provided?
It's Yiddish and the posh name is shm-reduplication
from wikipedia...
Latimer Alder suggests we third-raters discover The Appendix. He must have left active science a very long time ago to be unaware of the various appendices (starting from microprint and currently up to online deposit) that have been tried.
There are a lot of Puritans commenting here on how "Science" should be done. Mere practitioners have a hard time living up to their high standards.
The days of the self-funded scientist are long gone. So are the days when subscription to learned societies covered the cost of publishing globally. Scientific publishing is now a branch of commercial publishing- the editor chooses what papers are to be included, referees merely advise on scientific plausibility. Rightly or wrongly, granting bodies use past Impact Factor to assess future funding. As for open publishing on the Internet- it is, as Chou EnLai put it, "too soon to say".
@Latimer Alder
Dodgy Geezer asks the rhetorical question:
'"Is a published paper the best place to put reams of data about reagents?"
Perhaps, in addition to inventing the internet to overcome space limitations, we could invent 'The Appendix' (I realise 'professional scientists' will find such a concept beyond them) and put the relevant stuff in there.
......
Mr Alder has produced a ponderously sarcastic response to my comment which, unfortunately, does not hit the target he is aiming at. The rhetorical question was not, in fact, mine, as a moment's research would have indicated. It is a direct quote from the original paper, and is the main point of the piece.
If someone will not take the trouble to look up and read the original paper, I must reiterate my point more slowly and simply. The PEER paper does NOT indicate that reproducibility was affected, though that is, of course a possibility. It is part of a discussion about exactly WHAT level of data needed to be included in a published item, given that space was at a premium. Specifically, they searched for "exact product numbers for five types of biomedical resources", the implication being that using the same reagent from a different supplier or batch might affect the experiment. And so it might - but to what level of detail do you need to go?
This is an item of concern to professional scientists, but it is a detailed issue of publication standards and a million miles away from leaving out critical parts of your data manipulation because you can't justify them. The PEER paper is discussing the establishment of precise guidelines for (typically) chemical experiment specifications. The report comments:
"While suggestive of a trend, the study did not, however, report on how often a lack of specific information about experiment resources hindered reproducibility."
While we know that Climate Science is full of fraudulent attempts to claim proof where none exists, and there is a major problem about reproducibility of their findings, the PEER paper is talking about attempts to set standards in an area where publishers are confused about the levels of detail which are necessary. These are different things, and should not be merged together to make it look as if scientists across a wide range of disciplines are corrupt....
It's worse than we thought
http://www.reuters.com/article/2013/09/06/us-ipcc-climate-idUSBRE9850JM20130906
Friend of mine, a long time ago, suggested the following journal -
"The Journal of Pointless Studies in Luxurious Surroundings"
Sounds like we have it.
@dodgy geezer
You suggest that I have misinterpreted your earlier comment. That may be so, but it is undoubtedly true that you have misinterpreted mine.
I was not trying to suggest that the misbehaviour of climate scientists was widespread in other fields, and if you felt I was directing my 'third-rate' remarks there, then I apologise for the lack of clarity. It was not intentended to be read that way.
But there is an underlying point in that 'professional science' is - to all intents and purposes - a self-regulating field. There is no statutory body to oversee professional standards - such as we see in medicine or accountancy or dentistry or a whole host of others. Neither have there been many acts of legislation to determine how and what should be done. No doubt you would join me in applauding this freedom - allowing scientists to do their work unfettered.
But - and it's a big but - with that freedom comes the responsibility to regulate yourselves wisely and with the good of both the indivduals concerned ..and the wider public in mind.
And it seems ot me that science has been ill-served in recent years by the moot acceptance of poor standards by those leaders who should be encouraging ethics and integrity, not hiding behind convenient rocks like 'its not my field' and 'I can't comment on another's work'. Those are just cop outs unworthy of serious consideration.
And if the 'profession' really wants the public to retain their trust in science and scientists, it needs to do a lot more to show that it does not turn a blind eye to such dodgy dealings.
And to Sceptical Chymist who remarks 'There are a lot of Puritans commenting here on how "Science" should be done'
Then I plead guilty. Perhaps I was lucky to have studied at a good university in the 1970s under some practicing scientists who set store by maintaining their personal and professional integrity. And then to work in commerce where bad practice was not only frowned upon but where bad practitioners could - and occasionally did - end up in jail.
I make no apology for expecting high standards from those we entrust with a lot of our cash and a lot of freedom to work as they choose. Thse who find it too hard should perhaps find alternative employment. There is no more a God given right to do shoddy science than there is to do shoddy medicine or auditing.
@Latimer Alder
Hmm...misinterpretation seems to be the flavour of the month today!
Let us begin by enumerating our points of agreement. You are concerned at the poor standards displayed by many current climatology scientists? I agree. You are disappointed at the leadership shown by the authoritative bodies in this field? I agree. The Royal Society has been particularly appalling in this regard. You feel that the 'profession' (I prefer 'calling') ought to stop turning a blind eye to the 'science as advocacy' we have been experiencing? I also agree.
The only point I wanted to make was that this particular report was presented as a paper about yet another scandal of this type, in the life sciences. And I don't think it is. It is a paper indicating that some standards are dropping - not though intentional activism but due to pressure of publication space and a lack of agreement/understanding as to what constitutes essential information. And it proposes a standard to rectify this. I would say that this was evidence of science putting its own house in order quite effectively...
While I hope I share the high-minded aspirations of Latimer Alder and others, I am aware of the real-world pressures on scientists.
I was provoked into commenting on this thread because I found it particularly rich in the hostility to scientists that has recently been emerging at BH- even His Grace, normally a fair minded prelate, wrote waspishly at the top: "Unless you are an scientist, of course, in which case replication means being able to reach vaguely similar results using different, but equally obscure methods."
Dodgy Geezer uses the term scandal about a paper that only poses a sober question: How much information about complex reagents needs to be included in the primary publication? Of course, all data available to the experimenter should be recorded [and made available on request!], but even deposited data (appendices) have space limits in most journals.
Let us examine a point made in the PEER paper more closely. A complex "reagent" like a monoclonal antibody or established cell line, is special in that it is originates from only one source and might alter during propagation and storage. Thus the reagent used by one worker _could_ have different properties from a nominally identical reagent used by another. In the absence of any indications to the contrary, practitioners assume they are the same. If there is a hint that this is not the case, emails fly about warning colleagues of any doubt. To do otherwise is impractical.
Please don't turn into "science alarmists"- after all, man-made CO2 _might_ cause catastrophic global warming, and this new version of the Precautionary Principle could make doing science so much less fun that only the third-rate will bother.
I have just noticed in another thread that "scientivist" is not a typo but a neologism! Apologies to His Grace.