Bishop Hill Questions to ministers
Apr 9, 2013 Climate: Statistics Climate: Surface This is a guest post by Doug Keenan
Questions relating to the work of the Met Office on global warming are being put in the UK parliament, and the Met Office is refusing to answer them. Parliamentary Questions have a history going back centuries. Giving answers, or giving a valid reason for not answering, is required. The stand-off is yet to be resolved.
The Parliamentary Question that started this was put by Lord Donoughue on 8 November 2012. The Question is as follows.
To ask Her Majesty’s Government … whether they consider a rise in global temperature of 0.8 degrees Celsius since 1880 to be significant. [HL3050]
The Answer claimed that “the temperature rise since about 1880 is statistically significant”. This means that the temperature rise could not be reasonably attributed to natural random variation—i.e. global warming is real.
In statistics, significance can only be determined via a statistical model. As a simple example, suppose that we toss a coin 10 times and get heads each time. Here are two possible explanations.
- Explanation 1: the coin is a trick coin, with a head on each side.
- Explanation 2: the coin is a fair coin, and it came up heads every time just by chance.
(Other explanations are possible, of course.)
Intuitively, getting heads 10 out of 10 times is very implausible. If we have only those two explanations to consider, we would conclude that Explanation 1 is far more likely than Explanation 2.
A statistician would call each explanation a “statistical model” (roughly). Using statistics, it could then be shown that Explanation 1 is about a thousand times more likely than Explanation 2; that is, statistical analysis allows us to quantify how much more likely one explanation (model) is than the other. In strict statistical terminology, the conclusion would be stated like this: “the relative likelihood of Model 2 with respect to Model 1 is 0.001”.
A proper Answer to the above Parliamentary Question must not only state Yes or No, it must also specify what statistical model was used to determine significance. The Answer does indeed specify a statistical model, at least to some extent. It states that they used a “linear trend” and that the “statistical model used allows for persistence in departures using an autoregressive process”.
If you are unfamiliar with trending autoregressive processes, that does not matter here. What is important is that HM Government recognized, in its Answer, that some statistical model must be specified. There is, however, still something missing: is the choice of statistical model reasonable? Might there be other, more likely, statistical models?
(There is also a minor ambiguity in the Answer, because there many types of autoregressive processes. A related Question, from 3 December 2012, effectively resolved the ambiguity. The Answer to the Question stated that “Linear trends … are based on year-to-year variability around trends described as autoregressive (AR1) processes” [HL3706]. Other Answers, discussed below, confirmed that the process was of the first order.)
I found out about the Question (HL3050) put by Lord Donoughue via the Bishop Hill post “Parliamentarians do statistical significance”. I then discussed the choice of statistical model with Lord Donoughue. I pointed out that there were other models that had a far greater likelihood than the trending autoregressive model used by the Answer. In other words, the basis for the Answer to the Question was untenable.
Moreover, I had published an op-ed piece discussing this, and related issues, in the Wall Street Journal, on 5 April 2011. The op-ed piece includes a technical supplement, which describes one other statistical model in particular: a driftless ARIMA(3,1,0) model (again, unfamiliarity with the model does not matter here). The supplement demonstrates that the likelihood of the driftless model is about 1000 times that of the trending autoregressive model. Thus the model used by HM Government should be rejected, in favor of the driftless model. With the driftless model, however, the rise in temperatures since 1880 is not significant. In other words, the correct Answer to the Question (HL3050) might be No.
Lord Donoughue then tabled a Parliamentary Question asking HM Government for their assessment of the likelihood of the trending autoregressive model relative to the driftless model. HM Government did not answer. Lord Donoughue then asked a second time. They did not answer. He asked a third time. Again they did not answer. He asked a fourth time. Still they did not answer. He has now asked a fifth time. The answer is due by April 12th.
A Parliamentary Question that has been tabled in the House of Lords is formally answered by HM Government as a whole. In practice, HM Government assigns the Question to a relevant ministry or department. In our case, the Questions have been assigned to the Department of Energy and Climate Change; the designated minister is the Parliamentary Under Secretary of State, Baroness Verma. Verma obtains answers from the Met Office. The person at the Met Office with final authority is the Chief Executive Officer, John Hirst. In practice, Hirst delegates authority to the Chief Scientist at the Met Office, Julia Slingo. Thus, it is actually Slingo who is refusing to answer the Parliamentary Questions, with Hirst and Verma backing her (perhaps without thinking).
I have had a few e-mail exchanges with Slingo in the past. Slingo has never really addressed the issues that I raised, but instead replied largely with rhetoric and a display of gross ignorance about undergraduate-level statistics. For an example, see the Bishop Hill post “Climate correspondents”. Hence, I decided that trying to talk directly with Slingo about the Parliamentary Questions would be a waste of time. Instead, I tried talking with Hirst. I first e-mailed Hirst about this after the third refusal to answer the question from Lord Donoughue. The message included the following.
Last week, Lord Donoughue tabled Parliamentary Question HL6132, about statistical models of global temperature data. HL6132 is essentially the same as HL5359, which the Met Office refused to answer. The Met Office Chief Scientist does not have the statistical skills required to answer the Question; there is, however, at least one scientist at the Met Office who does have the skills—Doug McNeall. I ask you to ensure that the Question is answered.
Doug McNeall is a statistician. He and I have had cordial e-mail discussions in the past. In particular, after my op-ed piece in WSJ appeared, on 12 August 2011, McNeall sent me an e-mail stating that the trending autoregressive model is “simply inadequate”. Indeed, that would be obvious to anyone who has studied statistical time series at the undergraduate level. Note that this implies that a statistician at the Met Office has stated that the Answer given to the original Parliamentary Question (HL3050) is unfounded.
Hirst’s reply to my message was sent after the fourth refusal to answer the question, on 28 March 2013. It is as follows.
I would like to assure you that the Met Office has not refused to answer any questions. The questions you refer to were answered by Baroness Verma, Parliamentary Under-Secretary of State at the Department of Energy and Climate Change.
I note that in her response to HL5359 and HL6132, and a number of other questions from Lord Donoughue, Baroness Verma has offered for him to meet officials to discuss this and related matters in more detail.
My rejoinder is below.
I do not know whether your message is serious or just your way of telling me to get lost. In case of the former, some elaboration follows.
The question that Lord Donoughue has been asking requires the calculation of a single number. The calculation is purely arithmetical: there is no opinion or judgment involved (nor is background in climate needed). Furthermore, the calculation is easy enough that it could be done in minutes, by someone with the appropriate statistical skills. You could think of it as being similar to finding the total of a column of integers.
The number that Lord Donoughue is asking for is 0.001, according to my calculation. (Yes, it is that simple.) Lord Donoughue, though, would like the number calculated by an official body. He therefore tabled Parliamentary Questions asking HM Government for the number.
Lord Donoughue has now received Written Answers to four such Parliamentary Questions: HL4414, HL5031, HL5359, HL6132. None of those Answers give the number. Instead, the Answers make excuses as to why the number is not given. The main excuse seems to be that the number is not important. The importance of the number, however, is a separate issue: even if the number has no importance at all, the arithmetical calculation can still be done, and the number can still be given.
HM Government has been relying upon the Met Office, to supply them with the number; the Met Office has refused to do this. In other words, the Met Office has refused to answer the question—contrary to the claim in your message. What reason does the Met Office have for refusing to supply the number? The required time would be less than the amount of time that the Met Office has spent in refusing.
Parliamentary Questions have a history going back centuries. I do not have expertise in this area, but it is my understanding that HM Government is obliged to either provide an Answer to a Question or else give a valid reason for not providing an Answer. The refusal of the Met Office to supply the number would thus seem to be leading to a violation of a centuries-old parliamentary convention. Indeed, I have now talked with other members of the House of Lords and the Commons about this: there is real concern, and apparently also by parliamentary officials.
Lord Donoughue has now asked for the number a fifth time. The tabled Question is as follows (HL6620).
To ask Her Majesty’s Government … whether they will ensure that their assessment of [the number] is published in the Official Report; and, if not, why not.
The Answer is due by April 12th. My hope is that if the Met Office continues to refuse to supply the number, HM Government will get the number from elsewhere.
I have not received a reply to that. Additionally, I have been informed that Hirst is away this week and next; so there will be no reply from Hirst before the due date of April 12th.
The Met Office is obviously being highly obstructionist. The alternative, though, would be for the Met Office to admit that they do not have a statistical model that supports their claim that the temperature increase since 1880 is statistically significant. In other words, the alternative is for the Met Office to admit that the temperature increase might be reasonably attributed to natural random variation.
It is not only the Met Office that has adopted a position like this. The IPCC, in its most-recent Assessment Report (2007), used the same statistical model as the Met Office. The Assessment Report discusses the choice of model in Volume I, Appendix 3.A. The Appendix correctly acknowledges that, concerning statistical significance, “the results depend on the statistical model used”.
What justification does the Appendix give for choosing the trending autoregressive model? None. In other words, the model used by the IPCC is just adopted by proclamation. Science is supposed to be based on evidence and logic. The failure of the IPCC to present any evidence or logic to support its choice of model is a serious violation of basic scientific principles—indeed, it means that what the IPCC has done is not science. The failure implies that the claim that temperatures have been significantly increasing is unfounded.
Thus, the Parliamentary Questions tabled by Lord Donoughue undermine the primary basis for global-warming alarmism. The Met Office is trying for a cover up. In doing so, it is potentially risking a conflict with parliament.
It remains to be seen how matters get resolved. Under the rules of parliament, the person delegated with responsibility for a Parliamentary Question is the government minister who delivers the Answer. In our case, that minister is Baroness Verma. According to the Companion to the Standing Orders and Guide to the Proceedings of the House of Lords, §4.68 Ministerial Responsibility, “Ministers should be as open as possible with Parliament, refusing to provide information only when disclosure would not be in the public interest” and “Ministers who knowingly mislead Parliament will be expected to offer their resignation to the Prime Minister”.
Bishop Hill
Doug McNeall tweets that this article is relevant to the points Doug Keenan makes.
Reader Comments (69)
I have a rule of thumb that the more detailed the statistical arguments become regarding "significance", then the less likely it is that anything useful will be derived from them.
By analogy, the SSRI class of anti-depressants have only marginally established effects over placebo, but most people know they are probably a waste of time and money. A better placebo is needed.
So, if the ARIMA(3,1,0) model is so much better than the other one, then what aspects of the physics of climate does it capture?
link you fool,LINK! doh!
http://www.rawstory.com/rs/2013/04/08/climate-change-predicted-to-cause-more-in-flight-turbulence-for-passengers/
Could some one point me to a definition of a driftless ARIMA? The ARIMA bit I'm OK with, it's the driftless bit that I don't know about - I thought the I in ARIMA implied drift?
It is well known that mankind's CO2 emissions were not significant (common sense of the word) prior to the 1940s.
http://www.epa.gov/climatechange/ghgemissions/global.html
Therefore, mankind's global warming effect could not have been much in the first half of the subject 1880 to present period.
Serious question: What was the global warming during 1880-1946, and what was it during 1947-2012? Was there a significant difference?
Regarding the post from Doug McNeall, I replied in a comment there.
@ RichieRich Apr 9, 2013 at 12:08 PM
Given a statistical model, and data, statistical significance is straightforward to compute (at worst, via Monte Carlo simulations). This is something that everyone agrees on, as far as I know.
@ Heide de Klein Apr 9, 2013 at 12:58 PM
If we know that the coin is fair, then Explanation 1 cannot be a possibility.
@ Heide de Klein Apr 9, 2013 at 3:10 PM
If the likelihood of trending AR(1) relative to ARIMA(3,1,0) is very small, then that strongly indicates that trending AR(1) is failing to explain some substantial structural variation in the data—regardless of the physical plausibility of ARIMA(3,1,0)—and so trending AR(1) should be rejected.
@ JK Apr 9, 2013 at 3:35 PM
The Wikipedia article on ARIMA defines “drift”. An ARIMA process is driftless when the drift is zero. In fact, this might well be the ARIMA processes that you know; I just put “driftless” in for emphasis.
Well, from 1947 to 1997 there was global warming, but since then there hasn't been any. This might not be statistically significant, depending on your model, but there has to be a physical reason for it.
I suspect that with the advent of satellite measurements, it has become increasingly difficult to cool the recent past, hence the hiatus. We know the past was cooled, but we don't know by how much since all the data was disappeared.
In summary, the case for draconian and expensive measures to combat something we can't place a level of significance on is not sufficiently robust. Easy. Is that the right answer?
confusedphoton said:
Got it in one, until these crafty, dissimulating civil servants are de-funded [for heavens sakes - they've been defenestrated] - the lies will go on and on - as long as the [taxpayers] money rolls in.
I get the feeling asking this question is like asking Al Capone if he has filed his tax returns with accurate statements about his income, a simple question that lead to his imprisonment and loss of power over tax evasion. But is it this simple?
Are you saying that if the Met Office says that they use the "trending autoregressive" statistical model, they will no longer be able to say the post-1880 temperature rise is statistically significant, or that they have a "95%" certainty that anthropogenic CO2 is the cause?
It is clear that temperatures have risen since 1880, and more rapidly than averaged from 1880 during the 1975 to 1998 period. Which the average citizen would say is the point (though he conflates correlation with CO2 with causation etc. etc.). So what would the average citizen get from resolution of this dispute in the way that you propose, i.e. with an admission and number?
Not seeing headlines here, Capone didn't see his, either.
Apr 9, 2013 at 3:43 PM | Don B
" What was the global warming during 1880-1946, and what was it during 1947-2012? Was there a significant difference?"
I have been discussing this with Doug over on his blog. It was approximately 0.4C from 1880 to 1940, and 0.8C between 1880 and 2000. Although the rise from 1980 to 2000 was about 0.5C. (temperature fell after 1950 or so). My statistical knowledge is crap, so I can't get into the discussion on the type of statistices. What I have been asking Doug is if the 0.8C rise is significant (scientifially) over the 120 year period how would he describe the rise of 0.4C ove a 60 year period. He hasn't really given me an answer, or at least I don't understand his answer.
I can see how you would describe a rise in temperature as scientifically signifiicant say, if it went way beyond the bounds of any change that has occurred before, or beyond what you know about natural variability and the other activities in what I will call the biosphere. I want to try to understand what they expected the temperature to be in 2000 that made them declare 0.8C scientifically significant.
My hat is off to you, Doug Keenan, for your intelligence, energy, and resourcefulness in behalf of science and in the face of an unresponsive bureaucracy. This story has to hit the media sooner or later.
Bayesian methods cannot avoid begging the question.
Apr 9, 2013 at 10:14 AM | simon abingdon
Spot on.
I've read his blog and have had some interaction with him, but have come to the conclusion that [snip for tone - he] had anticipated running rings roung the ignoramouses who don't work at the Met Office. Big fail.
He spoke of scientific signficance, so I asked him what it was and how it applied to 1880 -1940. He described that as a hypothetical question. I've pointed out that there must be some criteria by which he judges 1880 - 2000 as scientifically significant rise in temperature and simply wanted to know what would happen if he applied the same criteria to 1880 - 1940. Still no response, although it's late and he might have more to do with his time than blog. Which he appears to do during he working day at the Met Office given the timing of his posts.
I'm not sure he has anything to add to our knowledge, he hasn't engaged with anyone and seems remarkably thin-skinned for someone in a scientific research where the failure rate makes even the most eminent and successful scientists extremely reluctant to dismiss even the humblest of people out of hand.
Apr 9, 2013 at 11:31 AM | geronimo ...
Forgive my ignorance, McNeal's statement to the effect that 0.8C increase in temperature since 1880 is "significant" seems to be an emotional response in he does not have a valid statistical model to confirm this "significance". Surely then, if there is not a 'mechanism' to describe the statistical significance of the temperature increase he is incorrect in describing the increase as "significant".
Given Paul Homewood's expose of the current state of the UK-MO understanding of climate, the only answer that McNeal is capable of providing is, "we don't know."
MacNeil's splitting of statistical and semantic hairs reminds me of Lord Rutherford's statement that 'if you need to use statistics to validate your experiment, you should have designed a better experiment'.
Great job. Since the subject at hand is statistical analysis, someone should throw this at them as well.
It completely refutes their claims, which are alchemy... Not science.
http://www.earth-syst-dynam-discuss.net/3/561/2012/esdd-3-561-2012.html
Doug McNeall (from e-mail to Doug Keenan):
A significance test attempts to answer the question “given that there was no anthropogenically driven global warming, what is the probability that we would see these temperatures?” This is interesting, but not really what we are looking for.
Doug (McNeall): with all due respect, I think that is exactly what we are looking for. If the probability of seeing the recent half century or so of temperatures with no AGW is not very small (for some value of "very"), then it is hard to make an argument that AGW must exist, and therefor that we should gamble our economic well-being to prevent it - and this, of course, pre-supposes both the arguments that CO2 reduction is the best response to reduce AGW, and that reduction is more cost-effective than mitigation
Apr 9, 2013 at 12:03 PM | Brent Hargreaves wrote
The PQ should be:"In declaring a 0.8C rise in temperatures since 1880, what temperature does the government consider current and what would it otherwise have been had greenhouse gases not risen since then?"
Could I suggest a slight change? "In declaring a 0.8C rise in temperatures since 1880, what temperature does the government consider current, what does it consider the temperature would otherwise have been had greenhouse gases not risen since then, and what is the scientific, logical and statistical basis for these conclusions?"