Bishop Hill More from Norfolk Police
Jan 17, 2012 Climate: CRU Climate: Russell FOI Readers may remember that I asked for Norfolk Constabulary's correspondence regarding the Russell inquiry - this was centred around the extraction of emails from the CRUBACK3 server by the police's forensic IT contractors, Qinetiq.
Although much correspondence was released, there were many redactions, few of which made any sense. I therefore launched an appeal and when this was (bien sur) rejected appealed to the Information Commissioner.
Although I have had no decision from the Commissioner, today I had another letter from the Constabulary:
We have engaged with the ICO throughout their consideration of your appeal.
As part of this engagement we have returned to the relevant third parties, the UEA/Muir- Russell Inquiry and the company Qinetiq, to again seek their views on whether information about the negotiation between them for the provision of a service should continue to be treated as information that attracted a duty of confidentiality. The parties advised that there are two significant factors that have come into play since the original response was made; the passage of time and a change in the commercial functions of Qinetiq. This has led to both third parties removing their objections to the release of information originally refused under section 41. Therefore, the Constabulary no longer has reason to apply the exemption and we can release the information. A further set of emails is attached.
The release is here. I don't see much of interest yet, although I need to check this to the first release. I don't recall that I was told that any emails were withheld completely. I'm also not sure that the gaps - the redactions - have now been filled in.
Reader Comments (49)
Why do they need to spend all that time and money trawling through the emails of Jones, Briffa, Osborne & Hulme? Do they suspect them of leaking FOI2009.zip?
Surely if they are going to pay for forensic IT support they should be focusing on analysing firewall logs, IDS logs etc.
The Russell inquiry wanted the police to get the emails for them. THe police had removed the servers for analysis.
Ah I see, I was thinking of the original police investigation into the leak.
FOIA did £55,440 worth of email filtering for free wow
I wonder if any of the redacted cc's pertained to UEA.
Will the appeal continue or will it now stop?
It looks to me like they decided to concede rather than losing on appeal and creating a precedent.
QinetiQ was asking for £742.50 to extract emails from the FOI2009.zip file.
Didn't they know there was a free online search tool, within a couple of days of the release, that could do it for nothing?
"Just for your information this amounts to well over 100,000 emails"
and FOIA still has 250K under wraps?
Suppose quite a few will be "get some milk on the way home" type?
Re: Steve McIntyre
While you can not see the email addresses you can see the structure of the addresses by hovering over the redacted part. This is the structure of all the redacted addresses.
xxxxxx@xxxxxxx.xxx
xxxxxxx@xxxxxxx.xx.xx
xxxxxxxx@xxxxxxx.xxx
xxx@xxxxxxxxxxxxx.xxx
xxxxx@xxxxxxx.xxx.xxxxxx.xx
xxxx.xxxxxxx@xxxxxxxxxx.xxx
xxxxxxxx@xxxxxxx.xxx.xxxxxx.xx
x.xxxxxx@xxxxxxxxx.xxxxxxxxxxx.xxx
x.xxxxxxx@xxxxxxxxx.xxxxxxxxxxx.xxx
They might have added or subtracted letters to/from the email addresses when replacing them with x'es or the above might be nothing more random collections of x'es
TerryS
I'd bet they did. Come on they started at £55K and dropped to £9k. The ICCER was a feeding frenzy. Everyone knew what it was about.
This document clearly identifies which "researcher's" emails and backups were retrieved from the servers ('all emails sent and received by Prof Phillip Jones, Prof Keith Briffa and Dr Tim Osborn').
So it is now absolutely clear that the Russell inquiry had the information to determine whether Jones had been deleting emails in response to FOI requests or not. Russell and Boulton avoided making this determination..
Hence the Jones, the UEA and their bought and paid for lackeys, Russell and Boulton, are corrupt, unethical, charlatans (or fools).
If the UEA or anyone in British science wished to clear this up - they could simply compare Jones' email archive and backups.
My understanding is that the the email systems at UEA used Endura.
If so it is simple to search.
These costings appear to be excessive.
ZT - we have good reason to believe that for CG1 the emails were extracted from themed inboxes for Briffa, Jones, Osborne and Hulme. Jean S spotted that for CG2 it appears that there are also emails that are probably from Tom Melvin's inboxes as well.
The following report from the UEA enquiry provided the result of the brief analysis of emails from Briffa, Jones and Osborne.
http://www.cce-review.org/evidence/Report%20on%20email%20extraction.pdf
For some reason that report did not cover Hulme
http://bishophill.squarespace.com/blog/2011/7/31/more-from-the-police.html
Don Keiller - "My understanding is that the the email systems at UEA used Endura."
Eudora methinks.
I am afraid this reminds me of the Marx Brothers Contract Skit from A Night At The Opera.
FOIA may generate more heat than light sometimes. I doubt the process is good for one's sanity over the long term though.
xxxx.xxxxxxx@xxxxxxxxxx.xxx
That's Sir M
UKP55,000 seems like small spuds in the context of the fees paid to various whitewashers. Just sayin'.
"Eudora"
I thought the problem was that they used Thunderbird, which Sir Muir had never heard of...?
xxx@xxxxxxxxxxxxx.xxx
That's Jimbo
TerryS Jan 17, 2012 at 4:15 PM
Didn't "professor" Jim Norton (he's only a visiting prof, not a real one with a university chair) say that because the emails used an unusual email system (Thunderbird?) it was not possible to extract them. If I recall, there were some sniggers at the capability of the Russel enquiry's IT "expert".
xxxxxxx.xxx.xxxxxx.xx is probably norfolk.???.police.uk
"If I recall, there were some sniggers at the capability of the Russel enquiry's IT "expert"." --Martin A
'Capability' is a function of objective. If the objective was to obfuscate, divert, conceal, dissemble, and confuse, then the expert was well chosen, innit?
TerryS said: "xxxxxxx.xxx.xxxxxx.xx is probably norfolk.???.police.uk"
norfolk.pnn.police.uk
@clivere
These files include two things. The server backups, and the email folders of Jones (and Briffa and Osborn).
By comparing these two pieces of data (server backups and Jones' folders) the police (assuming they were interested in detecting crimes) can determine if Jones was deleting information that had been requested under FOI.
The current situation is that the 'evidence' is being sequestered in a police station under the pretense that it has something to do with international hacking. Basically Acton, Boulton, and Russell hid the evidence of Jones' crimes or innocence in a police station!
You might think it would be the duty of the police or at least the many and various inquiries to prove Jones innocent by comparing the server backups with his email folders (but this would be naive, climatologically, of course).
ZT - my understanding is that breaches of FOI were ruled upon by the ICO not the "Independent"!!!!! review and that UEA were found to be culpable based on what was established from Climategate. However due to a technicality they could not be penalised for the breaches.
The Peter Sommer TOR was apparently to "identify email traffic which was not published on the various websites, but nontheless related to the same issues and might justify further investigation by the Independent Review".
In my opinion he was correct to point out it was a lot of effort to look at the emails but I have sympathy with the view that he should have tried harder to look for those emails that would have added to the context of some of the key published emails.
A summary of the UEA IT as it relates to Climategate based on what has been published. I have seen some people make the claim that because the enquiries were a whitewash then the published IT information must be wrong. However it aligns sufficiently well with what people have established from forensic details contained in the CG files that I am happy it is probably correct. In particular the great hope from a lot of people that the CG files were magically created for some activity associated with FOI is and always has been totally wrong.
As identified in this post at CA (which gave Steve McIntyre the excuse for some sport) UEA had a year earlier been proclaiming the success of their new secure IT access.
http://climateaudit.org/2009/11/28/uea-succeeds-in-quest-for-secure-it-access/
http://networking.cbronline.com/news/uea_succeeds_in_quest_for_secure_it_access_121108
(There is no view in the article whether it would have fully addressed CRU IT and the indication from the enquiry is possibly not. However the CRU IT would probably still have to exist within the framework of UEA IT and the introduction of secure IT access may have led to stresses and strains within interfacing IT functions. There is a clear indication of common internet connectivity and common email gateway.)
Interview carried out at UEA on 27th January 2010
http://www.cce-review.orgevidenceUEA-CRU_IV3_IT_Final_Formal1.pdf
3. In common with other areas of the Science Faculty, the CRU operated largely independent of the central IT functions of the University. Central IS had, in recent years , made significant efforts to better support the Science Faculty and some use of central facilities (such as the Storage Area Network) had been achieved. The University IS team did not provide desktop, remote access, hosting, database or software support to the CRU, nor any quality control or assessment. CRU had their own local arhitecture based on a mix of individual PC based and server based processing.
Internet communications for the CRU were however routed over the university network and through the university firewall.
The CRU had originally had no central backup arrangements for the individual researchers' PCs however Mike Salmon had introduced automated backup (using open source software) to a simple server held securely within the IS machine room. Jonathan Colam-French (Director Information Services) indicated that whilst the central IT function were aware of the existence of the CRU Backup Server, they had no knowledge of the nature of the information held on the server as it was managed from the CRU.
29 March Salmon response
http://www.cce-review.org/evidence/29%20March%20Salmon%20response.pdf
Q: What material (if any) beyond e-mails was contained on the CRUBACK3 server and may thus have been subject to unauthorised disclosure?
The entire "C:" partition of a Windows PC, or the "Users" directory of a Mac, or the "/home" directory of a Linux PC would be backed up, with certain exceptions:
A few people have secondary drives installed which were also backed up either in part or entirety.
Only one researcher elected to manage his own backups. All machines that were backed up to the server may have contained emails if the user used an email client that stored their email on their desktop machine, which has been the common practice in CRU.
UEA Central E-Mail Domain
http://www.cce-review.org/evidence/university%20archive%20emails%20enquiry%20120410_Final.pdf
All emails to the University are delivered to our email gateways. These are Unix servers which host the MX record for "uea.ac.uk", filter all emails for virus and spam (Can-it) and deliver them via Unix sendmail to the appropriate email service.
Departments do not run separate email services and all email services are run from the centre. We run two different email services; an Microsoft Exchange service for staff which uses the standard MAPI, IMAP and POP protocols and a Unix email service (EXIM/Dovecot) for students which uses IMAP and POP protocols. However it should be noted that a number of staff have requested that their email account be hosted on the Unix email service. This is mainly Science research staff and including CRU researchers.
clivere,
Just reading through what the Police have now released you get the idea that the Russell Review team were very keen to get their hands on the full back up. Beddington told the STC that Sir Muir was going to put the leaked emails into context - "we all know how things can be taken out of context". The Review Report stated "The presumption is that emails were selected to support a particular viewpoint".
At a public cost of £9,000 Prof Sommer had all of Jones', Briffa's and Osborn's emails 54 days before the Review Report was published. He was supposedly an expert with facilities. I, on the other hand, only have XP with M/S standard software and have not got past that crazy dog that searches files for particular words. On 5000 files it took me only minutes to find my name in about 100 emails and only a few more to find 2526.txt. It would only have taken Prof Sommer a few days to have found it assuming my name was the last search key he tried.
2526.txt shows us that rather than deleting what I had asked for Briffa and Osborn just moved it onto a memory stick. It is not unreasonable to suspect that is what they all did. So Acton is right to tell the STC that no emails were deleted and they can all be read. He just did not tell the whole story. It was still an r.19 offence and not just by Jones.
I do not know that Sommer found 2526.txt. But I'd bet dollars to doghnuts that he read some emails just to sure that he could do so consistently. Based on what I've seen you would not spend an hour randomly reading before concluding that disclosing then all was not going to restore the reputation of the CRU.
All the emails will eventually be returned to UEA - its still their server. If they do not release them all I just hope FOIA gives us the password to the rar file.
David Holland - I am sympathetic with the view that the enquiries went out of their way to avoid looking at areas where CRU could be heavily criticised. I dont know if the Sommer statement that it was too difficult to search was also an excuse to avoid looking at the emails for incriminating material. I would accept it is a possibility that was part of his motive.
The Muir Russell review said in their report:
'we have seen no evidence of any attempt to delete information in respect of a request already made.'
....which has been widely parroted by gullible people.
Probably Muir Russell (or was it Sir-ethical-Boulton?) meant to say:
We have evidence of deletion of information in response to FOI requests, but have managed to avoid looking at it, and we have hidden it in a police station for now.
(Eventually the drives will be unreadable - and the world can go back to thinking that the UEA and the CRU are populated by Nobel prize winning honest scientists).
ZT - your wrote QUOTE - and the world can go back to thinking that the UEA and the CRU are populated by Nobel prize winning honest scientists - UNQUOTE
That may or may not happen.
Have your read the recent posts by Nickolov & Zeller over at Tallkbloke's Talk Shop?
It's just possilble that what they have to say may somewhat change future events.
Like the whole AGW theory is wrong and is now in process of being proved to be wrong?
So perhaps future Nobel Prizes may be awarded to (gasp) skeptics?
Surely not?
Or ..........
AusieDan - as long as people publish their methodology - and do not rely on splicing together spurious indicators that happen to point in politically expedient directions - I'm sure that the truth will emerge.
Meanwhile the climatological cesspool at the CRU could still do with draining...
@David - I think that you are being extremely generous to Acton in saying that 'no emails were deleted and they can all be read' was an honest statement to parliament.
Emails were hidden (in a police station, on thumb drives, wherever) to avoid revealing the CRUs activities - it is as simple as that. With electronic information, hiding is equivalent to deleting.and 'can be read' needs to be qualified with 'if you know where I, Jones and Briffa have hidden the storage media'.
In my opinion, it is absolutely clear that Acton, Oxburgh, and the Russell inquiry team were complicit in hiding information and misleading parliament.
Rmember that Manns answer to the Penn inquiry asking if he had deleted any emails was to produce a memory stick with the emails on board. A very clever way of making sure an FOI request could not be satisfied but if the shit hit the fan they could still be produced showing no deletion, especially if the inquiries were desparate to show no wrong doing.
The astonishing thing about this affair is the sheer arrogance of these people.They knew by 1999 [the Tanzania meeting] that they were involved in a group which had set out to deceive the population and government by manipulating data. Also they were apparently aided and abetted in this by others in the UK scientific hierarchy which then set out to protect them when the Climategate I e-mails were released.
In view of the aim if the project, which is to impoverish the UK population by imposing very high energy costs, including non-security of supply with the attendant massive rise in crime and danger of disease in the cities for no reduction of CO2 and perhaps an increase, this would be a major crime of which the e-mails are prime facia evidence.
And just as Met Police collars are being felt now that the Murdochs have lost their power over government, will the fact that an ex-employee of News International, Neil Wallis, reportedly organised PR for CRU in this period, mean that members of the Norfolk plod might be arraigned?.
Acton: "This is Acton speaking from UEA. Am I speaking with the professor?"
Jones: "Yes. Good evening, Vice-Chancellor Acton."
Acton: "Please tell me your name."
Jones: "I'm Professor Jones, Vice-Chancellor."
Acton: "Jones? Listen Jones. There is a Select Committee taking place in London. Now you go with your models to the railway station. There is a train. You will climb on that train and go to London. You go into the committee room and then you will tell me how many skeptics there are. Is that clear? I'm recording this conversation, Professor Jones …"
Jones: "Vice-Chancellor, let me tell you one thing …"
Acton: "Speak up! Put your hand in front of the microphone and speak more loudly, is that clear?"
Jones: "In this moment, the decline is hiding …"
Acton: "I understand that, listen, there are MPs that are coming down and asking questions. You go up to London, get in that committee room and tell me how many people are still not convinced. And what they need. Is that clear? You need to tell me if there are skeptics, deniers or activists in need of assistance. And tell me the exact number of each of these categories. Is that clear? Listen Jones, that you saved yourself from the ICO, but I am going to … really do something bad to you … I am going to make you pay for this. Go to London, (expletive)!"
Jones: "Vice-Chancellor, please …"
Acton: "No, please. You now get up and go to London. They are telling me that in London there are still skeptics …"
Jones: "I am here with the climate models, I am here, I am not going anywhere, I am here …"
Acton: "What are you doing, professor?"
Jones: "I am here to co-ordinate the models …"
Acton: "What are you co-ordinating there? Go to London! Co-ordinate the models from the committee room. Are you refusing?"
Jones: "No, I am not refusing."
Acton: "Are you refusing to go to London, professor? Can you tell me the reason why you are not going?"
Jones: "I am not going because the warming has stopped."
Acton: "You go to London. It is an order. Don't make any more excuses. You have declared 'the science is settled'. Now I am in charge. You go to London! Is that clear? Do you hear me? Go, and call me when you are in London. My PR spin crew is there."
Jones: "Where are your PR spin crew?"
Acton: "My PR spin crew is on the case. Go. There are already skeptics, Jones."
Jones: "How many skeptics are there?"
Acton: "I don't know. I have heard of one. You are the one who has to tell me how many there are. Christ!"
Jones: "But do you realise it is dark and here we can't see anything …"
Acton: "And so what? You want to go home, Jones? It is dark and you want to go home? Get to that Select Committee using the Hockey Stick and tell me what can be done, how many skeptics there are and what their complaints are. Now!"
Jones: "… I am with my second in command."
Acton: "So both of you go up then … You and your second go to London now. Is that clear?"
Jones: "Vice-Chancellor, I want to go to London, but it is simply that the tree rings here … there are other models. Warming has stopped and is waiting …"
Acton: "It has been an hour that you have been telling me the same thing. Now, go to London. Go to London! And then tell me immediately how many skeptics there are there."
Jones: "OK, Vice-Chancellor."
Acton: "Go, immediately!"
We know CRUBACK3 is the backup server.
Anyway of finding out more about CRUWEB08?
@ Mac Jan 17, 2012 at 5:10 PM
RONFL, thanks for a bit of humour, classic.
Still not any the wiser what exhibit DAW/1 ie CRUWEB08 represents and I dont recall seeing it referenced anywhere else. As it contains emails I would guess at one of the researchers laptops.
At a guess, CRUWEB08 is probably the production email server so would do the live email for internal CRU users. That would probably hold the live message store and also allow functions like webmail access. CRUBACK3 then archives that. If that is how it worked, then it may also have been how their servers got compromised. CRU probably needed webmail or remote access for when it's people were offsite at conferences or just home working. Because that would have been publicly accessable, it's a potential entry point for a hacker who could have then jumped onto the CRUBACK3 server.
The disclosure docs do highlight a problem the police have though. The task order asks for something simple, ie copies of all the emails from both servers. That's possibly the wrong question to ask given subsequent correspondence says that's around 7TB of emails. The other questions asking for emails to/from an unamed "named individual" make more sense but Qinetiq's quote seems unreasonably high to decompress standard arhives and run what should be some fairly simple queries. As does the suggestion that it'd take 2+ months.
The ability for police investigators to parse email logs or call records to draw up a contact web and do some social network analysis seems to me to be an essential service and something investigators should be able to do quickly. Problem in this case seems to be with Qinetiq having custody of the data, they can charge what they like and may not be incentivised to offer their customers the best advice. But that's privatisation for you.
Atomic Hairdryer -re CRUWEB08 -thanks - plausible - not seen anything to confirm that but I will keep it in mind.
re the relationship with QinetiQ I will make the observation that the FOI material does refer to the police having found information of interest but they were not disclosing to UEA or the enquiry at that time what it was. The Sommer email extraction report also indicates he had significant access constraints to work under to ensure data protection Whilst the quotes appear pricey we dont understand the constraints and activities involved well enough to know if the charges are in reality reasonable.
The high charge for looking at FOIA2009.zip is probably self inflicted given that UEA should have done that themselves from the outset anyway!
I'd still suggest £55k to extract some emails and produce some contact reports is unreasonable, especially when it should be routine work for investigators to be able to know who has been emailing who. Tools to do this kind of datamining and SNA have been available from the likes of Autonomy for years.
I think the rest was an artifact of their being multiple, supposedly independent investigations. The police were conducting their own investigations into the reported crime, and any others they found along the way. Like the FOI violations. There's no reason why the police should disclose everything to the UEA, especially if the UEA were being investigated.
Sommer's requests were seperate to that and supposedly for the Russell report to investigate any impropriety. The police's involvement in that seem just to have been due to them having the relationship with Qinetiq who had custody of the evidence. Data protection concerns may have been more of a consideration for that inquiry given it was "independent", so not subject to some of the exemptions the police have for investigating crimes.
Some of the quotes may have ended up higher than they might have been simply because of the initial speculation that this was the work of some state actor. That may have resulted in the data being bumped up to Top Secret and explain the odd access restrictions mentioned by Sommer. Those restrictions seemed odd to me given the Russell report's descriptions of CRU's IT didn't look like it was designed to store or process classified data. Or, if CRU were handling classified data, perhaps part of the investigation is to determine why it wasn't properly protected. Working with classified data does up the costs, otherwise investigatory costs can appear high simply because they have to be prepared in a way that could stand up in court.
It's also interesting to me that the disclosures to date don't seem to show a search for a hacker, just a "person of interest" in the archives.
The requirement is "to, from, or linked to" a named person. To and from is relatively easy, but linked to is a lot harder. It would require reading every email.
Also, as clivere notes, data protection and respect for privacy apply. The police cannot publicise the outcome of any searches they do on your property unless it is relevant to their case, and their contractors will be subject to the same rule. They will have to read and redact any personal information from any emails they passed on to the university enquiry. There's no reason to suppose that the police don't have full access, at a far lower price.
FOIA has spent the past two years extracting just 5,000 emails. Tom Nelson has been reading them for weeks. Every one has to be read and understood in context, references to other people or their work identified, its relevance judged. Even people who know the background like Steve McIntyre are taking time to figure it all out - I don't suppose Qinetiq has the climate-debate expertise to match.
Qinetiq has an interesting history, http://en.wikipedia.org/wiki/Qinetiq
Nullius -
I read "to/from or linked to" as wanting to select those emails in which the individual's email address is present in either the "To:", "From", or "cc:" portions of the header. I wouldn't consider this task to be terribly difficult or time-consuming, having completed task 3.1 (recovery of the emails from the backups -- which presumably involves reformatting from whatever archival file format existed.)
I don't think that reading (& comprehending!) the emails was intended. That would, as you say, be a Herculean undertaking.
Re Nullius
To/From or linked to any named individual depends on interpretation, so a simple interpretation is any email containing that name and the context. The precise requirement may have been subject to some clarification before Qinetiq produced their quote. It also should not have taken much time, after all we're lead to believe that an external "hacker" managed to find the contents of the original FOIA file amongst the 7TB of data in only a few days. Especially when Qinetiq say:
"This will have to be a manual search as there are no keywords to allow the machines to run automatic searches"
Yet a "hacker" allegedly found some.
As for redacting personal information passed to the inquiry, there should be no need to do that given the inquiry was tasked to look into those people's activities.
Looks like Frank Swifthack has seen this post.
http://ijish.livejournal.com/45456.html
He is misreading the Mike Salmon account as
"This suggests that the entire e-mail archive of each user was backed up at regular intervals. Thus if (say) the backups were done once a month, and the e-mail archive contained messages from 1990 all the way to 2009, then each and every month the backup server in UEA's central IS would receive another copy of the entire 1990--2009 cache, possibly with some additions, deletions, and movements along the way.."
However this does not reconcile with the statement from 18th Dec meeting with IT Personnel which suggests an incremental backup
http://www.cce-review.org/pdf/MR%2018%20Dec%20final%20IT%20Personnel.pdf
"Configuration of back-up server was unfortunate as it did not remove deleted emails."
With regard to the earlier discussion / possibility that CRUWEB08 is a webserver it appears the CRU web server was taken offline quite early on (27 Nov) which may be consistant with it being taken away but I am still not clear if it would contain emails.
email 0626
Hi Tim,
It's gone back to 16 mins today.
I upgraded BackupPC last week. My guess is that they've changed the algorithm for
incrementals so it effectively did a full backup and then some. Once it did a real "full
backup" the new algorithm works properly so incrementals go back to a more manageable
size.
I'll be doing a further upgrade in a few days, so it may do it again. Fix would be to
force a full backup.
Mike
Tim Osborn wrote:
Hi Mike,
my cruto4 (Windows) machine incremental backups seem to have gone from 15-45 minutes
previously up to 281 minutes on Wednesday and been running since 10am today and still
going! Any idea what's happening?
So the open source software being used was Backuppc
http://backuppc.sourceforge.net/faq/BackupPC.html#overview
BackupPC is a high-performance, enterprise-grade system for backing up Unix, Linux, WinXX, and MacOSX PCs, desktops and laptops to a server's disk. BackupPC is highly configurable and easy to install and maintain.
For example, the Eudora email tool stores each mail folder in a separate file, and attachments are extracted as separate files. So in the sadly common case of a large attachment emailed to many recipients, Eudora will extract the attachment into a new file. When these machines are backed up, only one copy of the file will be stored on the server, even though the file appears in many different full or incremental backups. In this sense Eudora is a "friendly" application from the point of view of backup storage requirements.
An example at the other end of the spectrum is Outlook. Everything (email bodies, attachments, calendar, contact lists) is stored in a single file, which often becomes huge. Any change to this file requires a separate copy of the file to be saved during backup. Outlook is even more troublesome, since it keeps this file locked all the time, so it cannot be read by smbclient whenever Outlook is running. See the Limitations section for more discussion of this problem.
http://backuppc.sourceforge.net/faq/security.html
http://www.bbc.co.uk/news/world-15840562
"A hacker entered a backup server at the university and downloaded a file containing administrative passwords"
The "downloaded a file" looks plausible
http://backuppc.sourceforge.net/faq/security.html
An important security risk is the manner in which the smb share passwords are stored. They are in plain text. As described in the ``Setting up config.pl'' section in the documentation>, there are four ways to tell BackupPC the smb share password (manually setting an environment variable, setting the environment variable in /etc/init.d/backuppc, putting the password in __TOPDIR__/conf/config.pl, or putting the password in __TOPDIR__/pc/$host/config.pl). In the latter 3 cases the smb share password appears in plain text in a file.
If you use any of the latter three methods please make sure that the file's permission is appropriately restricted. If you also use RCS or CVS, double check the file permissions of the config.pl,v file.
Further thoughts on the use of BackupPc by CRU and its role in the Climategate release.
http://backuppc.sourceforge.net/faq/BackupPC.html#overview
Features include:
A clever pooling scheme minimizes disk storage and disk I/O. Identical files across multiple backups of the same or different PC are stored only once (using hard links), resulting in substantial savings in disk storage and disk writes.
Optional compression provides additional reductions in storage (around 40%).
A powerful http/cgi user interface allows administrators to view the current status, edit configuration, add/delete hosts, view log files, and allows users to initiate and cancel backups and browse and restore files from backups.
No client-side software is needed
Flexible restore options. Single files can be downloaded from any backup directly from the CGI interface. Zip or Tar archives for selected files or directories from any backup can also be downloaded from the CGI interface.
Looking at the notes of the following meeting released as part of the evidence for the MR enquiries.
18th Dec meeting with IT Personnel
http://www.cce-review.org/pdf/MR%2018%20Dec%20final%20IT%20Personnel.pdf
"JCF - backup was compressed, so whoever did this would need software to restore files. Could have been a staff member in CRU or someone using a CRU computer. Much more difficult/sophisticated to do this externally. What was published on website included an "FOIA folder" - which was not a replication of what was on the back-up server. This is something which had been put together in this way by whoever published the data".
For me this note was always one of the more interesting IT related comments from the enquiry and when matched against the specification for BackupPc gains added interest.
BackupPC works using a pooling scheme as well as compression. This is described in more detail
http://backuppc.sourceforge.net/faq/BackupPC.html#some_design_issues
Pooling common files
To quickly see if a file is already in the pool, an MD5 digest of the file length and contents is used as the file name in the pool. This can't guarantee a file is identical: it just reduces the search to often a single file or handful of files. A complete file comparison is always done to verify if two files are really the same.
Identical files on multiples backups are represented by hard links. Hardlinks are used so that identical files all refer to the same physical file on the server's disk. Also, hard links maintain reference counts so that BackupPC knows when to delete unused files from the pool.
For the computer-science majors among you, you can think of the pooling system used by BackupPC as just a chained hash table stored on a (big) file system.
BackupPC supports compression. It uses the deflate and inflate methods in the Compress::Zlib module, which is based on the zlib compression library (see http://www.gzip.org/zlib/).
The $Conf{CompressLevel} setting specifies the compression level to use. Zero (0) means no compression. Compression levels can be from 1 (least cpu time, slightly worse compression) to 9 (most cpu time, slightly better compression). The recommended value is 3. Changing it to 5, for example, will take maybe 20% more cpu time and will get another 2-3% additional compression. Diminishing returns set in above 5. See the zlib documentation for more information about compression levels.
BackupPC implements compression with minimal CPU load. Rather than compressing every incoming backup file and then trying to match it against the pool, BackupPC computes the MD5 digest based on the uncompressed file, and matches against the candidate pool files by comparing each uncompressed pool file against the incoming backup file. Since inflating a file takes roughly a factor of 10 less CPU time than deflating there is a big saving in CPU time.
The combination of pooling common files and compression can yield a factor of 8 or more overall saving in backup storage.
Given the extracted CG emails released so far are limited to a maximum of 5 specific researchers it looks more likely that the person who released the emails would have done so using the BackupPC CGI interface rather than a more difficult and probably random search through the file pool. In order to use the CGI interface it appears a user would need to be recognised in the BackupPc hosts file. For me this makes it likely that the person who released the emails had possession of a PC with the appropriate CGI interface already established (or cloned) and managed to get in as a legitimate user (or administrator) with password.