This is a guest post by Shub Niggurath.
;)
In a self-publicized Environmental Research Letters paper, Cook et al claim to have performed a climate consensus-hunting literature survey they call "the most comprehensive of its kind" using "the largest sample". The comprehensive nature of the survey is important to the author and his group. For instance, close associate Lewandowsky writes:
There has been evidence in the peer-reviewed literature already that more than 95 out of 100 climate scientists agree on the basic premise that human greenhouse gas emissions are warming the planet. [...]
But until now, tools for the visualization of that evidence have been limited.
This is where the new study by Cook et al plays such a particularly important role: Going beyond previous surveys of climate scientists, Cook et al. performed a systematic review of the massive literature on climate change.
In a nutshell, they used a scientific search engine (ISI Web of Knowledge) to gather all papers published on ‘global climate change’ or ‘global warming’ between 1991 and 2011. This search returned a mind-boggling 12,000 papers
Several folks have looked at the Cook paper, assuming its basic soundness. But does the paper do what it says it does?
The climate change literature comprises well over hundred thousand articles and books. Cook et al’s strategy was to focus on papers directly related to "global warming" or "global climate change" in Web of Science. Here's how they describe it:
In March 2012, we searched the ISI Web of Science for papers published from 1991–2011 using topic searches for 'global warming' or 'global climate change'. Article type was restricted to 'article', excluding books, discussions, proceedings papers and other document types.
A Web of Science search performed following the authors' description to the letter actually returns 30,940 entries, not 12,464. Excluding the 'Arts and Humanities Citation Index' (A&HCI), this becomes 30,876. This is when search phrases are not enclosed in double-quotes (i.e., 'global warming' instead of "global warming").
Scopus is an academic database covering technical, medical, and social science disciplines. Surprisingly, when Scopus is searched using the correct search phrases, a total of 19,417 entries are retrieved. A Web of Knowledge search returns ~21,488 records. These figures are 7473 records (Scopus) and ~9544 records (Web of Knowledge) greater than what Cook et al eventually analysed.
These results make plain that a large body of relevant literature has been excluded by the authors in their study.
Conclusion
The Cook et al numbers are somewhat replicable, only if search is limited to the Science Citation Index and Social Sciences Citation Index databases in Web of Science. Presumably, this made the job of classification easier. Contrary to claims however, this makes their literature search incomplete. It is neither 'comprehensive' nor produces the "largest" possible data set. The finding of incomplete search has further implications as it affects all conclusions drawn in the paper.
H/T: Richard Tol, who made the original discovery on Scopus.