Seen elsewhere
Twitter
Support

 

Buy

Click images for more details

Recent posts
Recent comments
Currently discussing
Links

A few sites I've stumbled across recently....

Powered by Squarespace
« The names you need to know | Main | Decarbonisation amendment defeated »
Tuesday
Jun042013

Climatologists raise the shutters again

About a year or so ago, the Met Office and CRU announced the release of the CRUTEM4 temperatures series. As John Graham-Cumming noted at the time, they were positively gung-ho in their enthusiasm for transparency:

Given the importance of the CRUTEM land temperature analysis for monitoring climate change (e.g. Trenberth et al. 2007), our preference is that the underlying station data, and software to produce the gridded data, be made openly available. This will enhance transparency, and also allow more rapid identification of possible errors or improvements that might be necessary (see e.g. the earlier discussion of homogeneity adjustments in the SH).

The data was made available at the time, but, according to the Met Office, the code would follow soon after:

The code required to produce the CRUTEM4 fields and timeseries will soon be made available. Note that code previously released for CRUTEM3 cannot be used to exactly reproduce CRUTEM4 temperature series due to changes in processing methodology.

That's what they said at the time. However, if you look at the equivalent page now, it reads as follows:

Note that code previously released for CRUTEM3 cannot be used to exactly reproduce CRUTEM4 temperature series from these files due to changes in the processing methodology used.

I guess this means that the code is no longer going to be released.

According to JGC, the change to the web page took place after May 17th. I wonder if there is any connection to the Keenan article?

PrintView Printer Friendly Version

Reader Comments (28)

I posted this earlier. Probably relevant here.

This code in R fetches Met Office data from this page.

base<-"http://www.metoffice.gov.uk/hadobs/indicators/";library(installr);
url<-"http://www.metoffice.gov.uk/hadobs/indicators/11keyindicators.html"
download.file(url, destfile="metoffice.html", quiet = FALSE)
tree<-htmlTreeParse("metoffice.html", error=function(...){}, useInternalNodes=TRUE, encoding="UTR-8", trim=TRUE)
links<-xpathSApply(tree, "//a/@href")
d<-paste(base, links, sep="");d<-d[3:79]
dwnldata<-function(x){
name<-file.name.from.url(x)
download.file(x, destfile=name, quiet = TRUE)
}
lapply(d, dwnldata)

Jun 4, 2013 at 10:19 PM | Registered Commentershub

Could be that there are issues with the latest update CRUTEM.4.2.0.0. code/methodology? CRUTEM4 monthly numbers taking far longer to produce than previous issues. Also at least one obviously spurious result has been posted. Has since been revised/corrected/adjusted after being brought to the attention of RB on a BH discussion board.

Following subsequent notation has been posted:-

"Update: a problem in routine updates of the CRUTEM4 resulted in a large amount missing data for January 2013 in CRUTEM.4.2.0.0. We recommend that users requiring data for January 2013 download an updated copy of the CRUTEM.4.2.0.0 data from the downloads page."

http://www.metoffice.gov.uk/hadobs/crutem4/

Jun 4, 2013 at 10:42 PM | Registered CommenterGreen Sand

The operative adjective is incorrigible.

Pointman

Jun 4, 2013 at 10:42 PM | Unregistered CommenterPointman

I wonder how many plagues will be required before the code is set free?

:)

Jun 4, 2013 at 10:47 PM | Unregistered CommenterJames

The underlying code has been long known:

CRUTEM4=CRUTEM3+rand()*(year()-2011)/10

Jun 4, 2013 at 11:04 PM | Registered Commenteromnologos

Perhaps Harry might get a job there and put together a readme file with lots of code as well as comments, and which could be widely shared in due course. He must have been pretty fed up with the work quality and environment at CRU, so even the Met Office might look attractive to him.

Jun 4, 2013 at 11:06 PM | Registered CommenterJohn Shade

Harry might have taken all the code out and just left the comments in, since that was the only way to get CRUTEM4 to do anything meaningful.

Jun 4, 2013 at 11:25 PM | Unregistered CommenterConfusedPhoton

"our preference is that the underlying station data, and software to produce the gridded data, be made openly available"

Maybe after the 'barbecue summer' forecast, what they meant was the 'griddled' data.

Jun 4, 2013 at 11:27 PM | Unregistered CommenterJesus Green

"... due to changes in the processing methodology used." = new adjustments to the temperature data.

Jun 4, 2013 at 11:31 PM | Unregistered CommenterDrcrinum

Hot griddled comments, Alive! Alive, oh!
==============

Jun 4, 2013 at 11:51 PM | Unregistered Commenterkim

The most disappointing thing is that when the actual CRUTEM3 code leaked out because it was mentioned in one of the CRU emails it was nicely written and it was very easy to spot the odd bug that myself and another person had seen: http://blog.jgc.org/2011/11/anatomy-of-crutem3-bug.html

Hopefully this is just a temporary blip and the Met Office (and others) will release code as a matter of course.

Jun 4, 2013 at 11:55 PM | Unregistered CommenterJohn Graham-Cumming

I commented a few days ago that their "talking point"now, on both sides of the Atlantic, is basically, "we don't need no stinkin' data--the science is settled." This post doubles down on that, adding that "you don't need no stinkin' data either, so we won't bother giving it to you."

Jun 5, 2013 at 12:09 AM | Unregistered CommenterHarry Dale Huffman

I assume that you mean the article which is titled, "how Scientific is Climate Science?" At URL:
http://www.informath.org/media/a41.htm

Jun 5, 2013 at 12:42 AM | Unregistered CommenterCC Squid

I recall that James Annan made an acerbic post about this at the time when the announcement was made about "septics" (his term) having to put up or shut up. Do they really want sceptics to fall into line? Or is this deliberate sabotage? or.....maybe incompetence...and I guess James Annan is happy that he lives in Japan and is funded there where they seem to tolerate serial losers.

Jun 5, 2013 at 1:13 AM | Unregistered Commenterdiogenes

Be prepared for this shutter as well
They haven’t processed further for some days as I anticipated a few days ago.
http://ocean.dmi.dk/arctic/icecover.uk.php
Its not going their way that’s all I can say it happens every time ice goes up like this. Expect a major downward adjustments in coming days. The team cannot accept this data!

Jun 5, 2013 at 2:05 AM | Unregistered CommenterEliza

They're just following the basic precept of climatology - Jones' First Law:-

Why should I make the data available to you, when your aim is to try and find something wrong with it?

Jun 5, 2013 at 8:38 AM | Registered CommenterFoxgoose

A book on investing pointed out that, if a company's annual results are late in being published, it is a very bad sign "Bad figures always seem to take longer to add up than good figures".

In the case of the MO, "Bad" = "does not show evidence of rapid global warming".

Jun 5, 2013 at 9:13 AM | Registered CommenterMartin A

There is a text file on the Met Office Crutem site that still has the old wording:

Code will soon be provided to enable users to make gridded fields and calculate a global average temperature anomaly time series from the station data provided here. This code is to be released under an Open Source licence that is contained as comments in the code. By running the code you indicate your acceptance of the licence.

Note that revisions have been made to the gridding and time series calculation procedures used in construction of CRUTEM4, as described in Jones et al. (2012). Code provided for CRUTEM3 cannot reproduce the CRUTEM4 results.

Jun 5, 2013 at 9:22 AM | Registered CommenterPaul Matthews

Martin A "A book on investing pointed out that, if a company's annual results are late in being published, it is a very bad sign "Bad figures always seem to take longer to add up than good figures".

In the case of the MO, "Bad" = "does not show evidence of rapid global warming"."

Back around 2008/9 when I used to eagerly await each months results as a new way to highlight to warmists that it wasn't warming, I certainly noticed that "bad" (as in low) results would take up to 1-2 weeks longer to come out.

But there was one February's result when I just waited and waited and waited and I think it was nearly a month late before we got the result out showing a massive drop in the monthly temperature (probably 0.3C). It was the coldest month in 14 years ... as I naively tried to get the press interested in the story. Only to find the Met Office had spent that month preparing a blitz of pro-warming non-science so that by the time the the figure came out the press were sick and tired of the subject and not willing to print another article.

Perhaps what was even worse, was that slowly over the next few years that February dip disappeared and the only way I could now know when it was would be to go and see if I still have the statistics for the period.

Jun 5, 2013 at 10:17 AM | Registered CommenterMikeHaseler

The Met Office still plans to release the code.

Jun 5, 2013 at 1:11 PM | Registered CommenterRichard Betts

I remember thinking at the time that "our preference" seemed to leave a lot of wriggle room.

For example, as Foxgoose alludes to above, Phil Jones may 'prefer' to release his data, but this is overridden by a preference for it not being criticised by Steve McIntyre.

Has not the Met previously used commercial and/or third party agreements to deny FOIA requests? I am sure they would have 'preferred' to release that information.

Jun 5, 2013 at 1:55 PM | Unregistered Commentermichael hart

The Met Office still plans to release the code.

Well, release it then. It appears to be sufficiently complete to be in use, so what's the problem?

Although I'd love to know how the output can be validated to any reasonable level of precision, on the basis that last week's Inshore Waters Forecasts for the West of Scotland couldn't even get the current conditions right most of the time. The actual forecast was complete bollocks.

Jun 5, 2013 at 2:16 PM | Unregistered CommenterNW

Richard Betts, if you get a few minutes could you please take a look at the following?

Previously on this thread I posted the notation on the HadCRUT4 page regarding 4.2.0.0 Jan 13 data.

I am puzzled by the explanation; the original issue was with how the CRUTEM4 numbers affected the HadCRUT4 calculations. Not with the CRUTEM4 numbers themselves.

HadCRUT4.1.0.0 Jan 13 was +0.432C made up from CRUTEM4 at +0.891C and HadSST3 at +0.292C.

HadCRUT4.2.0.0 Jan 13 was +0.378C made up from CRUTEM4 at +1.182C and HadSST3 at +0.292C.

So with HadSST3 remaining the same, the puzzle was how an increase of +0.291C in CRUTEM4 could result in a decrease of -0.054C in HadCRUT4?

There may well have been a problem with CRUTEM4 data but I am at a loss to see how that could be responsible for the above?

The "new" "corrected" 4.2.0.0 set of numbers for Jan 13 referred to in the MO notation are now:-

HadCRUT4.2.0.0 Jan 2013 is +0.450C made up from CRUTEM4 at +0.935C and HadSST3 at +0.292C.

Still puzzled, could you please ask somebody to explain where I am going wrong?

Jun 5, 2013 at 3:13 PM | Registered CommenterGreen Sand

It would seem the MO has some previous in this general area:

Turns out the Met Office completely screwed up the stratospheric temperature data that all the atmospheric circulation modelers have been back-testing to since the 1980s. Thompson et al. 2012:

A new data set of middle- and upper-stratospheric temperatures based on reprocessing of satellite radiances provides a view of stratospheric climate change during the period 1979–2005 that is strikingly different from that provided by earlier data sets. The new data call into question our understanding of observed stratospheric temperature trends and our ability to test simulations of the stratospheric response to emissions of greenhouse gases and ozone-depleting substances.

What was wrong with the original processing of the satellite data? Nobody knows:

The methodology used to generate the original Met Office SSU data remains undocumented and so the climate community are unable to explain the large discrepancies between the original Met Office and NOAA SSU products highlighted here.

Shades of the Climategate “Harry read me” file, and this was only just discovered. All the modelers have been using bogus data to try to calibrate their models. Go back to square one guys.

http://wattsupwiththat.com/2013/02/12/uv-shift-in-the-leaked-ipcc-report-more-inversion-of-the-scientific-method/

Jun 6, 2013 at 12:29 PM | Registered CommenterJohn Shade

"The methodology used to generate the original Met Office SSU data remains undocumented.."

Presumably they were worried that somebody might find something wrong with it...

Jun 6, 2013 at 12:59 PM | Registered Commenterjamesp

It would be good if it could be communicated to the general public that scientists are not as they might be led to believe reading temperatures etc. off a dial which keeps going up, but are in fact using a variety of data which all has to be "processed" in complicated ways to give an answer heavily influenced by what those scientists think the answer should be.

They almost can't help it - it is what they were taught as students; the best marks go to those whose experimental results best match the textbook answer. (I have seen this done when the experiment was incapable of producing the results.)

Jun 6, 2013 at 1:00 PM | Unregistered CommenterNW

Richard Betts -
Unless there is some proprietary code involved, any reasonable modern software development methodology will allow for publication of the contents of a particular version with little more than the press of a button, or a few lines of a command script. I've been involved with million-source-line programs which are required to archive all the relevant source code for each released version; it isn't hard if any forethought has been given to the problem.

If the MO are unable, you could ask someone well-qualified in both software and climate such as Steven Mosher to arrange the matter -- probably cost no more than a few thousand pounds. Unless there's some commercially-protected code of course, in which case it was unwise to promise a code release.

Jun 6, 2013 at 1:33 PM | Registered CommenterHaroldW

'still plans'. Heh, stillborn plans. Richard, your credibility dies with the Met Office. You've been used.
==============

Jun 7, 2013 at 5:09 AM | Unregistered Commenterkim

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>