Bishop Hill ‘Landmark consensus study’ is incomplete
May 27, 2013 Climate: other This is a guest post by Shub Niggurath.
;)
In a self-publicized Environmental Research Letters paper, Cook et al claim to have performed a climate consensus-hunting literature survey they call "the most comprehensive of its kind" using "the largest sample". The comprehensive nature of the survey is important to the author and his group. For instance, close associate Lewandowsky writes:
There has been evidence in the peer-reviewed literature already that more than 95 out of 100 climate scientists agree on the basic premise that human greenhouse gas emissions are warming the planet. [...]
But until now, tools for the visualization of that evidence have been limited.
This is where the new study by Cook et al plays such a particularly important role: Going beyond previous surveys of climate scientists, Cook et al. performed a systematic review of the massive literature on climate change.
In a nutshell, they used a scientific search engine (ISI Web of Knowledge) to gather all papers published on ‘global climate change’ or ‘global warming’ between 1991 and 2011. This search returned a mind-boggling 12,000 papers
Several folks have looked at the Cook paper, assuming its basic soundness. But does the paper do what it says it does?
The climate change literature comprises well over hundred thousand articles and books. Cook et al’s strategy was to focus on papers directly related to "global warming" or "global climate change" in Web of Science. Here's how they describe it:
In March 2012, we searched the ISI Web of Science for papers published from 1991–2011 using topic searches for 'global warming' or 'global climate change'. Article type was restricted to 'article', excluding books, discussions, proceedings papers and other document types.
A Web of Science search performed following the authors' description to the letter actually returns 30,940 entries, not 12,464. Excluding the 'Arts and Humanities Citation Index' (A&HCI), this becomes 30,876. This is when search phrases are not enclosed in double-quotes (i.e., 'global warming' instead of "global warming").
Scopus is an academic database covering technical, medical, and social science disciplines. Surprisingly, when Scopus is searched using the correct search phrases, a total of 19,417 entries are retrieved. A Web of Knowledge search returns ~21,488 records. These figures are 7473 records (Scopus) and ~9544 records (Web of Knowledge) greater than what Cook et al eventually analysed.
These results make plain that a large body of relevant literature has been excluded by the authors in their study.
Conclusion
The Cook et al numbers are somewhat replicable, only if search is limited to the Science Citation Index and Social Sciences Citation Index databases in Web of Science. Presumably, this made the job of classification easier. Contrary to claims however, this makes their literature search incomplete. It is neither 'comprehensive' nor produces the "largest" possible data set. The finding of incomplete search has further implications as it affects all conclusions drawn in the paper.
H/T: Richard Tol, who made the original discovery on Scopus.
Reader Comments (32)
Can we all agree now to waste no more time on this heap of garbage? I rather thought we had already done so.
Can I heartily second Jack Savage's plea? Attention only makes Cook and his pals feel even more self-important.
The problem with not challenging Cook is the next paper will cite this paper - ideally a paper completely refuting Cook (and Lew) is the answer
A number of scientists whose papers were used in the "study" have already said that their paper was mischaracterized as being pro-AGW. This paper by Cook et al is embarrassing and a joke. It is hard to believe crap like this gets published but unfortunately, it is not hard to believe that crap like this is given credence and authority by AGW true believers.
It's not a matter of 'we' ignoring Cook when the Washington Post highlights his research.
=================
Whether it is True Believers of the Climate Change Religion such as Cook or civil servants at the DECC or at the Met Office, there are an awful lot of people out there determined to ensure the perpetuation of the myth.
None of them are going to quietly admit defeat and walk away.
May 27, 2013 at 3:51 PM | mangochutney
I think you underestimate the academic social science world.
All the authors cited by Cook and Lewandowsky will cite them in their next paper.
They all derive apparent legitimacy from each other.
The sole intent of the paper was to grab all those "97%" headlines - which it managed to do. As the Skeptical Science "leak" made clear, the "consensus project" is more about marketing than analysis.
The real scandal is that a supposedly reputable scientific journal could publish dross like this paper, and thus allow itself to be made the tool of a partisan website in this way.
More con census than consensus.
A gigantic double logical fallacy - argumentum ad verecundiam and argumentum ad populum.
What an utter waste of space!
When are they going to accept that 'consensus' is political, not scientific?
Is this not exactly the same modus operandi as the previous "consensus paper. This time they started with about 30000 possibles (see above), thinned them down to 12000 by some unknown algorithm. Thinned these by taking extracts, again by some unknown method. Extracted 36% of what was left and finished with 97% of the detritus agreeing with them as long as they did not look too closely. Last time it was about 7000 scientists of whom few replied, eventually thinned out to 72 out of the last 75 agreeing with them. Are they so thick that they really think there is any science or legitimacy in these convoluted machinations or are they just totally corrupt? It sure ain't rocket science.
97% of doctors said stomach ulcers were caused by stress.
97% of climatologists in 1974 said the planet was the verge of the next ice age.
97% of sane people think Mr. Cook is comedy gold.
IMO Here's a reason why it's worth challenging Cook's junk:
http://judithcurry.com/2013/05/27/president-obama-calls-out-the-climate-change-deniers-in-congress/
"If I were wrong, then one would have been enough!" — Albert Einstein, commenting on the book 100 Authors Against Einstein.
Which shows you that a consensus of 100 well paid academics can be overturned by one patent clerk, if the patent clerk is right.
Here's the puzzle of the week: Rearrange the following letters to make the name of a scientist:
WHAT LYSENKO SPAWNED
One problem is that the bar is set so low when it comes to the MSM.
The 'progressive' journalists will gladly print anything which has the correct politico-scientific stance, so third-raters like Cook know that however slapdash and untenable their paper, it will still get wide coverage; it's like feeding jam to a baby.
@AAA - Lots of studies show that CAGW is wrong. Here's a couple:
Scafetta 2013 recently published:
"the solar activity increase during the 20th century contributed at least about 50% of the 0.8 °C global warming observed during the 20th century"
and regarding Canty et al 2013 also recently published:
"due to "prior neglect of ocean circulation" of the natural ocean oscillation the Atlantic Multidecadal Oscillation [AMO] ... the authors calculate that climate sensitivity to CO2 is about 38% less than claimed by the IPCC.
If something like 88% of warming last century was due to the Sun and the oceans then not much is left for CO2 and everything else.
You might want to read some more science. There are hundreds of similar peer reviewed papers which have come out in the last few years now that the IPCC's intimidation in breaking down.
Earlier this morning, Dana N tweeted that they had not used the Web of Science (as the paper says) but rather the Science Citation Index.
The main point is that Cook et al. used a sample, rather than the population, and that the sampling properties have not been investigated.
The Web of Science omits many journals, but not randomly so. It has a strong bias towards establishment journals.
Richard
Web of Knowledge includes Inscope, a database it describes as a "comprehensive index to the global journal and proceedings literature in physics, electrical/electronic engineering, computing, control engineering, mechanical engineering, production and manufacturing engineering, and information technology". A good proportion of these areas/journals are likely excluded from WoS.
WoK includes MEDLINE, which is not covered by WoS. If you go to Pubmed and search for "global warming" or "global climate change" you get >4000 entries. These papers are likely excluded as well.
Global climate change is by nature an interdisciplinary area of scholarly work. Clearly, the authors excluded thousands of articles that fully qualify to be included and analyzed. On the other hand, 177 papers on ethanol are included.
From the paper :
"In March 2012, we searched the ISI Web of Science for papers published from 1991–2011 using topic searches for 'global warming' or 'global climate change'. Article type was restricted to 'article', excluding books, discussions, proceedings papers and other document types."
Via Richard Tol above:
"Dana N tweeted that they had not used the Web of Science (as the paper says) but rather the Science Citation Index."
Was this a Peer Reviewed Tweet?
//
btw - an overdue "chapeau" to Shub, Richard and others for having the stomach to tackle this rubbish.
If just one scientific paper holds within it the basic elements from which a true grasp of climate science can be gleaned that would be a more important than the continuous cry to uphold any consensus. It really does not matter if it sides with the consensus view of the topic or not, what matters it that it is correct both in its theory and practical application.
IMO all this consensus nonsense is just PR for more unreasoned 'me too' researching.
Not banned yet - here's another reason for challenging Cook et al. I'm an Australian taxpayer and my hard earned folding is funding this crap.
I hope Cook's latest excrescence burns well. I'm going to need it to keep warm with this spring weather and my fuel bill as it is.
'The debate is over'...
So it may be, but that doesn't alter the fact that reality doesn't reflect the papers - no matter how many there are...
To provide a bit of background on what has actually been done:
Three major databases, Science Citation Index Expanded (SCI-E), Social Sciences Citation Index and the Arts and Humanities Index constitute 'Web of Science', which is a widely used academic database. The parent company that holds Web of Science has a product 'Web of Knowledge' which additionally includes Inspec and MEDLINE. Inspec is physics and engineering oriented, and Inspec is huge - they claim they hold 13 million entries. Medline, which is searchable online is a subset of Pubmed, both of which are repositories of biomedical literature though a significant portion is purely biology and biologic 'basic-science' related. Scopus is an Elsevier-owned product that is known to have even greater coverage in several areas, compared to Web of Science. It matches Web of Knowledge in coverage.
Of the above, from the 8 different possible permutations, as per Tol's Twitter conversation with dana1981, they carried out their search on one database - SCI. They mis-reported this as 'Web of Science' in the paper.
Going by the FAQ they put out, look at the search for “climate change” in Web of Science. Cook et al they say they found 43,650 papers. Searching Scopus with the same phrase returns 55,606 articles. A search in Web of Knowledge returns 63,213 entries. Web of Science brings up 46,797 papers. As can be seen, *all* these numbers are in the order of thousands greater than what Cook et al report. They are all with the same conditions: 1991-2011 range, English language, document type ‘Article’, published in Journals.
This is evidence for a systematic bias/deficiency in the paper. Their searching misses swathes of literature that are fully qualified to be included in the study, but are not, simply because the authors restrict their academic literature database searches.
Great post Shub
Please remember to keep kicking them while they are down :P
Dani has article over on grauniad basically anyone who challenge concensus is cherry picking schill troll for big oil (again). Usual Sels are bashing each other over head with words they don't undestand. Out-lying climatists are manning defenses and, once again, it takes the debate no-where.
Global Warming seems to be having a hard time these days. The heady days of new papers proving without doubt that Global Warming is an immediate danger seem to have become rather thin on the ground, and in their place a new topic of discussion has arisen. This can best be described as a 'Call to the Faithful' - an exhortation not to lose heart as the science collapses around their ears. Running questionable surveys attempting to show that the hypothesis is still valid is one example of this type of behaviour. But can this opinion be backed up with evidence?
I have just conducted a short Cook-type survey to examine this phenomenon. The methodology was as follows:
1 - find an independent presenter of Global Warming news stories. I picked 'Climate Debate Daily' - a site which provides one pro and one anti item each day.
2 - examine and caregorise the 'pro-global warming' stories into three headings, using the abstracts provided. I picked the following categories:
a) - a story providing New Data on global warming (typically reports of technical papers)
b) - a story emphasising Solidity of Belief in global warming (typically reports of pro-global warming activity)
c) - Other Stories (often comment on political or industrial activity)
Note that the New Data do not have to support Global Warming theory - they just have to be stories providing new information. In practice, most of the stories under this heading actually indicated that the threat was smaller than had been assumed.
The results for the most recent 30 are as follows:
Following the Cook methodology, I dropped the 'Other' figure. The percentages then become (rounded to my error bars):
Thus it is shown that the Global Warming industry spends three times as much effort on preventing people leaving the faith as it does on showing that the faith is correct. Which supports the thesis at the beginning of this piece...
@ZDB/AAA
Matey, I gave you a couple papers falsifying your statement. Ante up. As I stated upthread I know of hundreds more. Takes just one to falsify a hypothesis.
You realise don't you that declaiming arrogantly "that there's not a single published study showing anything else" without addressing the examples in a civilised fashion just says to everyone you're not able to use the scientific method?
And if you can't support your statement with science, why should anyone believe you?
Cook covers the period 1991-2011. Hypothetical. A scientist publishes a 1991 paper based on work he/she started in 1986, just after Al Gore released "Earth in the Balance". That's a guide to how sophisticated climate science was then.
Our scientist does not publish again because he/she realises that the 1991 paper was wrong. Yet, by Cook's methodology, the paper adds positively to the claimed consensus.
Plainly, the original author now does not. How many such papers are there? There seems to be no Cook attempt to estimate change of direction with time.
....................................
It's not just change of heart. It goes to the heart of the way that science progresses. What would have been the point of spending huge sums for new satellites with new instruments interpreted by new supercomputers, if the science was indeed settled, static, and in consensus in 1991? The mere fact of massive spending on new gear is almost certain proof that many senior scientists and decision makers did not believe the data nor the consensus. They wanted better figures. Sure, they could have wanted them to make the CAGW case stronger; but they could also have wanted to show it was weaker. Again, Cook's survey does not seem to differentiate this factor. Those senior people who approved the new projects were possibly not authors of papers discussing the later results, so they could have missed the Cook literature count. There is also a factor of disproportion. One of those unpublished approvers might have the scientific worth of 10 naïve junior authors who were counted.
.........................
Next, buddy Lewandowsky asserts that "Of all peer-reviewed papers expressing a position on human-caused global warming, 97-98% endorsed the facts that the Earth is warming due to greenhouse gas emissions."
If you go down this page of the Bishop's to the Doug Keenan articles, you find a very important admission from the UK Met Bureau to a Peer questioning from the House of Lords. The Met response is phrased as "That being the case, the rises in temperature over the last two centuries and over the last decades of the twentieth century, look like nothing untoward. The global warming signal has not been detected in the temperature records." This is remarkably like the oft-quoted Einstein condition, that "it takes only one paper to prove me wrong."
........................
I feel compelled to add these comments because I'm an Australian scientist who would not like to see these two, Cook and Lewandowsky, become Aussie science poster children instead of those behind the good and serious science being done here. Neither has the learning to comprehend the hard science that is good and honest, so they should shut up and stop fantasising about the abstract motivation and psychology of hard scientists. You have to be one to walk the walk.
How telling that the main claim to fame is that the American President has touted the study.
In the next paper the claim will be "197% of the scientist agree" ?