My family and me.
[This post got extensively re-written (you can tell that, cos it has a title that doesn't fit its URL :-) after I realised that I, too, had been fooled by the septic FUD. Oh dear. I've stopped now: you can read on without fear that the words will change under you.]
The septics are trying to pretend that there is a spat between the Swedes (SMHI) and CRU, but this is just smoke-n-mirrors. Lets quote the final letter first:
With reference to the current debate regarding, amongst other things, access to climate data we have found that our letter to you dated 21 December 2009 unfortunately have rendered bad publicity both to SMHI and to the climate research community. We understand now that our response to your request forwarded by UK MetOffice 30 November 2009 may have been misinterpreted, maybe due to the fact that the formulations may have been a bit harsh. Our response was based on your information that it was likely that the version held by you would most likely differ from our current holdings. It has never been our intention to withhold any data but we feel that it is paramount that data that has undergone, for instance, homogenisation by anyone other than SMHI is not presented as SMHI data. We see no problem with publication of the data set together with a reference stating that the data included in the dataset is based on observations made by SMHI but it has undergone processing made by your research unit. We would also prefer a link to SMHI or to our web site where the original data can be obtained.
That is from SMHI and is dated 4th March. So: Jones asks SMHI if he can release their data (via the UKMO, 30th Nov 2009; apologies for dodgy source). They say no (21st Dec 2009). He tells people that SMHI has said no. This looks bad, so SMHI changes their mind, as long as the data gets a disclaimer as to its source and processing. All is well, perhaps.
[Update: apparently some Swedish folk are watching GA, but they are doing it in Swedish :-). This one I lke, though.
There is More from Max Andersson. He makes an interesting point - that if you actually *read* the transcript, most of the quotes attributed to Jones come from Acton:
Professor Acton: Unfortunately, several of these countries impose conditions and say you are not allowed to pass it on, so there has just been an attempt to get these answers. Seven countries have said "No, you cannot", half the countries have not yet answered, Canada and Poland are amongst those who have said, "No you cannot publish it" and also Sweden. Russia is very hesitant. We are under a commercial promise, as it were, not to; we are longing to publish it because what science needs is the most openness.
Some license agreements here.
Also ClimateWTF.
Just to make it clear: despite what misc septic blogs are saying it is *not* true that Dr. Jones asserted that the weather services of several countries, including Sweden, Canada and Poland, had refused to allow their data to be released - see the transcript. I think Max A is the first to notice this. The lesson, again, is not to actually believe anything the skeptics say without verifying it first.
-W]
Except in the denialosphere, where nonsense abounds. So CA repeats
PRESS RELEASE Stockholm March 5, 2010 Climate scientist delivers false statement in parliament enquiry It has come to our attention, that last Monday (March 1), Dr. Phil Jones, head of the Climatic Research Unit at the University of East Anglia (CRU), in a hearing with the House of Commons Science and Technology Committee made a statement in regards to the alleged non-availability for disclosure of Swedish climate data. Dr. Jones asserted that the weather services of several countries, including Sweden, Canada and Poland, had refused to allow their data to be released, to explain his reluctance to comply with Freedom of Information requests. This statement is false and misleading in regards to the Swedish data. All Swedish climate data are available in the public domain. As is demonstrated in the attached correspondence between SMHI (Swedish Meteorological and Hydrological Institute), the UK Met Office and Dr. Jones (the last correspondence dated yesterday March 4), this has been clearly explained to Dr. Jones. What is also clear is that SMHI is reluctant to be connected to data that has undergone "processing" by the East Anglia research unit. STOCKHOLM INITIATIVE Göran Ahlgren, secretary general
So this is great FUD. GA is nothing to do with SMHI, he is just stirring up trouble (and if you looked at early versions of this post, you'll have seen that it took me a while to work that out).
In fact, SMHI say:
SMHI are developing its own web site where you can find data from a number of stations, including the stations in the list enclosed with your letter, downloadable for non-commercial purposes, please see http://data.smhi.se/met/climate/time_series/html/essential20.html and http://data.smhi.se/met/climate/time_series/html/vov20.html. The datasets are for a shorter period, but the web site is under development and more data will be available for download in the near future.
which doesn't sound like "all data available" in fact it sound like "not all data are available". So as expected GA is talking nonsense. Who *is* GA even? I don't know; judging from this he is just another std.septic (We the undersigned, being qualified in climate-related scientific disciplines... is the usual danger flag; and yes, when you look, GA is a professor of organic chemistry); and the other usual suspects are there Timothy F. Ball, PhD, environmental consultant and former climatology professor (no he wasn't).
The SMHI also say, in their response to Jones's original request:
Given the information that the version of the data from the SMHI stations that you hold are likely to differ from the data we hold, SMHI do not want the data to be released on your web site.
which is a bit odd, ah, but is explained by this other letter (again, dodgy source, sorry): the data has been homogenised and QC'd. Err, as you'd expect, really.
The site their letter points to - http://data.smhi.se/ looks very amateur. They haven't even turned off auto-indexing, go there and you get:
[DIR] hyd/ 04-Feb-2010 07:54 - [DIR] met/ 03-Feb-2009 11:34 - [DIR] oce/ 22-Jul-2009 08:45 - [ ] fil.zip 05-Feb-2010 10:15 158M
Environment
Comments
Wow. It's almost impressive how badly they (septics) mangle the truth. If the qualified yes arrived on March 4th, when did Jones lie about them having said no (which they obviously did prior to March 4th)?
They really do think he has a time machine, don't they?
Posted by: pough | March 6, 2010 3:45 PM
Hard to sift through things, but I think I figured out a timeline:
Nov 11, 2009: Jones asks for permission to release homogenized data
Dec 21, 2009: SMHI says no, we are eventually going to release it ourselves
Mar 1, 2010: Jones claims SMHI said no
Mar 4, 2010: SMHI clarifies statement, allowing publishing of data with clear mention of alterations and links back to SMHI for original data (which isn't yet up)
Mar 5, 2010: Septics claim Jones knew on Mar 1 the contents of the mail dated Mar 4
Mar 6, 2010: Rest of world eagerly awaiting photos of green sky from septics, future cricket scores from Jones
Posted by: pough | March 6, 2010 3:52 PM
it can be brought back further, though I'm not going to plow through stuff and get the exact dates.
Remember that when McIntyre's FOI was rejected on appeal, the rejection letter said that though they couldn't release proprietary data, they (UEA) intended to work with the national agencies and to try to get permission to redistribute that data.
This explanation as to why they had to reject the FOI, combined with the good-faith statement that they were going to work to try to improve the situation, would've satisfied most people.
OK, nose held, I've descended briefly into CA and, dripping with slime, got this from the appeal rejection letter:
This was sometime after the end of September, unfortunately there's no date on the note, but it's consistent with CRU having pinged the Swedes on November 11, where pough's timeline picks up.
Posted by: dhogaza | March 6, 2010 4:05 PM
A poster who goes by "Rattus Norvegicus" (frequents a variety of places) made the following catch ...
The Swedes are putting up data on their own website, as was stated in the first letter.
Take a looky-looky at the license you agree to when you download data from that website:
No confidentiality agreement my a**.
Posted by: dhogaza | March 6, 2010 4:10 PM
> dodgy source
Ya never know what's going to be found later on at a WFT link, things can change; has anyone asked the original source to acknowledge the material and provide a URL from that source directly?
Posted by: Hank Roberts | March 6, 2010 5:08 PM
"STOCKHOLM INITIATIVE" should have set off the alarm bells. Its like my economist buddy said: if its from "The Freedom Institute" or the "Liberty Foundation" or anything with a name that sounds like that, its going to be BS.
Posted by: bigcitylib | March 6, 2010 5:24 PM
When I first read it, I googled Stockholm Initiative and clicked on their about page. From that point on I new it was denier spin.
I think after you see the same type of half truths and distortions time after time it makes you suspicious. Unfortunately I don't want to have to be suspicious with everything I read. That is kind of depressing, but clearly now necessary.
Posted by: Kevin Meaney | March 6, 2010 5:40 PM
dhogaza #4:
The Hadley website has an even more restrictive agreement. Their data is available only to academics working towards a published paper. After the paper is done, the data is to be erased. This data destruction ( even if done by Phil) has no effect: Hadley still has the original.
Posted by: John McManus | March 6, 2010 6:43 PM
Hmmm... Lets think about that confidentiality agreement. Data is publicly available on a website, for anyone to download. Anyone, no filtering. SMHI states that you agree not to try to make money off of their data or give their data to anyone else. That is like me putting my house number on my front door and then yelling at everyone who drives down the street that my address is confidential.
You wanted to go out of your way to point this out as though it is meaningful in any way? Really?
Posted by: bill | March 7, 2010 1:06 AM
I think this post (final version) has still got it wrong. Jones does ask if he can release their data. But the reply on Dec 21 does not say no. Nothing of the kind. Instead, it says that Jones does not have their data anymore. They do not want him releasing what he has and claiming that it is SMHI data. That is clearly obvious. I'll say it again. They do not state that Jones cannot release their data. They very clearly state that Jones no longer has their data and they don't want anyone confusing what he has with their data. Here is the important line:
||Given the information that the version of the data from the SMHI stations that you hold are likely to differ from the data we hold, SMHI do not want the data to be released on your web site.
They don't say "You cannot share our data". All this talk of timelines is obfuscation. The communication of Dec 21 is pretty clear?
So, does Jones accurately report this? Not at all. And the further dissembling here is disconcerting. Jones says that SMHI told him that he could not release their data. That is misleading at best. Think about it. Jones got the data from them originally. He either still had a copy of it (and that copy would match what SMHI had), or the only thing he had left was the results of his data manipulation. SMHI did not want anyone to think that Jones data manipulation was their data. So they told him not to release it.
Now, its possible that under the strictest reading of the letter of the law, Jones did not commit fraud or any crime. But I can't see how what Jones did could be considered honest or open or ethical. He did everything possible to prevent anyone from knowing what the communication really said or that he no longer had a copy of the data that SMHI sent him that he could release.
Granted, this is a hypothetical, but does anyone really think that SMHI would have asked Jones not to release the data if they thought that he still really had their data in the first place?
Posted by: bill | March 7, 2010 1:23 AM
Göran Ahlgren has a degree in organic chemistry, but as far as I know he has no credentials in climate science.
Stockholmsinitiativet is the most active denier* group in Sweden. The only climate scientist attached to it is the retired paleoclimatologigist Wibjörn Karlén. Basically, they just translate the standard denier stuff into Swedish for local consumption and give journalists someone to turn to when they want to "cover both sides of the controversy".
* They hate being called deniers, but when temperatures contrary to their expectations turned out to be above average the last couple of months the response was to dig out Essex&McKitrick; and deny that Earth even has a mean temperature, so I think the name fits.
Posted by: Thomas Palm | March 7, 2010 2:02 AM
bill, it is more than hypothetical: it is outright wrong. During the enquiry Canada was taken as example of those countries not wanting CRU to release their data. In the statement by CRU, it is explicitely mentioned that Canada wants people to go to the CANADIAN repository to get the data. Same goes for the SMHI, as we now see.
And if you look at the message from the SMHI in #4, you will see that Jones is *still* not allowed to transfer the data from SMHI, especially if he doesn't do anything with it!
Posted by: Marco | March 7, 2010 2:06 AM
bill:
You need to learn when to give up. Phil Jones asked SMHI for premission to distribute the CRU version of their data. He could not have asked them for permission to distrubute the SMHI version because he did not have it. SMHI said no. That's it. End of story.
Posted by: rocco | March 7, 2010 2:44 AM
Dear Thomas,
I, for one, prefer to be called a "denier" rather than "skeptic" because the word "skeptic" indicates that I am genuinely open-minded about the question whether AGW is nonsense. While I am a "skeptic" about many general questions - it's a pillar of science to be skeptical - I am not really open-minded about the key AGW issues: the evidence is rather overwhelming that there's nothing to be worried about.
Concerning this story, it seems to be clear. The Swedes have always allowed Jones to publish any of their data because they're public domain. They just didn't want to be associated with his "adjustments". Jones has deliberately "misunderstood" what they were clearly saying, so that he could blame someone else for his own secrecy (which is needed for him to hide his mess and incompetency).
[You seem to be making things up. There is no evidence at all that the Swedes have "always allowed Jones to publish any of their data because they're PD". If you do have this evidence, please make it public -W]
I don't claim that the Swedes are saints but when it comes to the hiding of information and obfuscation, Phil Jones' CRU has surely been in the top ten (or bottom ten) of science places in the world.
Best wishes
Lubos
Posted by: Luboš Motl | March 7, 2010 4:04 AM
Lubos, you are indeed a denier:
"The licensee does not own nor authorized to redistribute, sell, assign or otherwise transfer data products or documentation without further processing to third parties unless the parties have received written permission from SMHI."
The SMHI explicitely demands no one publish their data, in particular when NOT processed(!), unless SMHI gives written permission. CRU has just (=March 4) received permission to distribute PROCESSED data, but with caveats.
Posted by: Marco | March 7, 2010 5:35 AM
Here's what I posted at CA earlier today (awaiting moderation, let's see if it gets through)...
________________________________________________________
The license on Dataserier 1961-2008 from SMHI clearly states that data may not be distributed (3.2)
November 30th 2009
Request made by UEA to make data more widely available.
December 21st 2009
Request rejected.
"Given the information that the version of the data from the SMHI stations that you hold are likely to differ from the data we hold, SMHI do not want the data to be released on your web site."
March 1st 2010
Parliamentary hearing.
March 4th 2010
Clarification by SMHI.
"We understand now that our response to your request forwarded by UK MetOffice 30 November 2009 may have been misinterpreted, maybe due to the fact that the formulations may have been a bit harsh."
Jones isn’t psychic. SMHI recognised that they have probably given the wrong impression.
Posted by: J Bowers | March 7, 2010 7:58 AM
I for one cannot take Lubos seriously. He's just stirring the pot with contrarian out-of-the-box takes on life, such as his hilarious "I would rather be called a denier than a skeptic" reasoning and his ending "best wishes". It's obvious, every sentence is tailored to annoy you in some way. I for one will just enjoy it as I enjoy the bitter taste of coffee.
Posted by: Cthulhu | March 7, 2010 10:34 AM
The proper description for Lubos is obnoxious (another word occurs here, but would be banned in the boathouse).
Posted by: Eli Rabett | March 7, 2010 10:50 AM
I think people are getting tripped up by the free aspect of the data. Free to download is not the same as free to redistribute.
Much more disconcerting is the rambling insanity of your comment.
Wha? I've never seen such a clear case of Morton's Demon in action before.
Posted by: pough | March 7, 2010 11:43 AM
My post #4 makes it clear it is not public domain. It is available subject to the terms stated there.
Posted by: dhogaza | March 7, 2010 11:48 AM
More on Lubos and his role (following Lindzen) in the "no warming since 1995" meme here:
http://deepclimate.org/2010/03/02/round-and-round-we-go-with-lindzen-motl-and-jones/
More on Goran A coming soon.
As far as I know, it's very rare for ClimateAudit to repeat an WUWT post (the other way around is more common). So McIntyre clearly thinks he has a winner here. Sheesh ...
Posted by: Deep Climate | March 7, 2010 12:01 PM
Stockholm Initiaitve shows people involved, including:
"Maggie Thauersköld, Copywriter and advertisement project leader. Owner of the blog "The Climate Scam". Maggie is coauthor with Lars Bern to the book Chill-out, the truth about the climate swindle.”
Advisor Peter Stilbs is an NMR physical chemist who ran “Global Warming – Scientific Controversies in Climate Variability,” Stockholm, SE, 09/11/06-09/12/06
gamma.physchem.kth.se/~climate
gamma.physchem.kth.se/~climate/speakers.htm
Speakers included Baliunas, McIntyre (oddly labeled U of Toronto), Singer, Soon, and an all-start cast of others from Europe. The conference reached the expected conclusions: no warming since 1998, expect global cooling, cosmic rays might be important, any warming is most likely natural, CO2 is not really important, carbon taxes or cap-and-trade are bad.
Stilbs also signed the APS2009 Petition ... so he is well-experienced in these things.
Fortunately for Sweden, post-glacial-rebound stays ahead of SLR for a good while.
Posted by: John Mashey | March 7, 2010 12:16 PM
Wow, pough. Excellent job of quoting selectively. That only works if my original words are SOMEWHERE ELSE. If people are too lazy to go look at what I really wrote somewhere else, your selective quotes do the trick. But what I wrote is right here, on the page.
If you ask me if you can date my sister, and I say that I don't have sister, so no you can't, what do you communicate to the world of our conversation? That I don't have a sister, or that I told you that you couldn't?. You went way out of your way there to not include a couple sentences of mine.
[Misrepresentations deleted - W]
Posted by: bill | March 7, 2010 1:42 PM
As post #4 clearly shows, you are not allowed to redistribute the raw data. Thus the FOI requests for such raw data (not just from Sweden, but other countries with similar restrictions) was properly rejected.
Bill, your (ahem) looseness with the facts isn't likely to convince anyone with read at oh, a 4th grade level.
Posted by: dhogaza | March 7, 2010 2:29 PM
umm "who reads" I mean ...
(must ... use ... preview ...)
Posted by: dhogaza | March 7, 2010 2:32 PM
http://www.realclimate.org/index.php/archives/2010/03/a-mistaken-message-from-iop/comment-page-3/#comment-165010
Brian Dodge quotes the language restricting redistribution.
Posted by: Hank Roberts | March 7, 2010 2:46 PM
> Pulitzer
Read the documentation at the Pulitzer site. The nominations are for, and the award is made for, the _writing_ -- the book, the newspaper series. The award is "given to" the writers, but the writing, the book or the newspaper series, wins the award. Look at the wording in the "Controversies" section and you'll realize you're just flogging horseburger long beaten by people looking to confuse. But you probably knew that, eh?
http://www.pulitzer.org/historyofprizes
Posted by: Hank Roberts | March 7, 2010 3:00 PM
That's for "The European Centre for Medium-Range Weather Forecasts" website, which maintains a data set aggregating data from a whole bunch of European countries.
The restrictions there are similar to those at the SMHI site that holds swedish data only.
I suppose someone could put together a laundry list of sources and restrictive license agreements, but in reality only one example is needed to prove that the UEA hasn't been lying about their existence.
Posted by: dhogaza | March 7, 2010 3:15 PM
I keep getting deleted, but I'm not sure why?
[You've said the same thing once. If you want to keep saying the same thing again, you'll need to get your own blog -W]
Posted by: bill | March 7, 2010 4:37 PM
Oh, and while we are being explicit, lets also not get even more confused with our time machines... That new website that they put up, does have a restriction about redistribution. But that isn't where Jones got his data. So, the restrictions on that website don't really have anything to do with the current conversation. You can assume that the original agreement where Jones did get his data had the same restriction, but I don't know why you would. If you do, care to share a source or at least explain why?
As of March 1, the only thing we know is that SMHI told Jones not to share his data because it wasn't SMHI data anymore.
Posted by: bill | March 7, 2010 4:42 PM
Göran Ahlgren is secretary general of "Stockolmsinitiativet" a Swedish climate-denial organisation based on the classic mix of academics, providing scientific cred and pseudoscience for the public, and bloggers and lobbyist, spreading disinformation and lies.
We in Uppsalainitiativet have been exposing their frauds and misdeeds since autumn 2008: http://uppsalainitiativet.blogspot.com/
Posted by: Anders M | March 7, 2010 5:34 PM
Hmm, maybe we should have something like a list of all the denialist organisations, so that people don't get confused when something like the Stockholm Initiative comes up?
Posted by: rocco | March 7, 2010 5:50 PM
A little more, in one palce:
Thauerskold, Maggie (Sweden)
http://www.theclimatescam.com
http://www.theclimatescam.com/about
“I was never particularly interested in economics. Therefore, I became an economist. Although only on paper. My current daytime job is in an ad agency.”
http://www.stockholminitiative.com/eng/about-us/who-we-are
“Copywriter and advertisement project leader. Owner of the blog "The Climate Scam". Maggie is coauthor with Lars Bern to the book Chill-out, the truth about the climate swindle.”
http://www.heartland.org/full/24843
“Swedish blogger Maggie Thauerskold will broadcast portions of the International Conference on Climate Change at her Web site, http://www.theclimatescam.com”
http://www.theclimatescam.com/2009/03/08/international-conference-on-climate-change she attended
Activities: Heart2009#2
Organizations: StockInit
People: Stilbs
Posted by: John Mashey | March 7, 2010 5:59 PM
36 bill: "Once again, if Jones got his data from that website, we can assume he is bound by the agreement from that website. But if he didn't get the data from that website, there might be a different agreement...He didn't have the SMHI data to share anymore, and he even admitted it in the letter he sent to SMHI."
How many SMHI's are there in Sweden? How old was the data? How many years do pretty much every single scientific research establishment on Earth say primary data should be kept for?
[Since I've deleted several of Bill's posts (and therefore the replies to them) this will probably be out of sequence now :-). Anyone who cares can read the comment policy -W]
Posted by: J Bowers | March 7, 2010 6:37 PM
have a look at this,
http://maxandersson.blogspot.com/2010/03/climate-sceptics-are-wrong-about-phil.html
Posted by: Cthulhu | March 7, 2010 6:59 PM
Cthulhu, that's very interesting. Apparently, Phil Jones was lying when Edward Acton said Sweden refused to let them redistribute the data. So not only can he travel through time, he can also speak through others!
Phil Jones is magic!
Posted by: pough | March 7, 2010 8:01 PM
@bill:
http://www.cru.uea.ac.uk/cru/data/availability/agreements.pdf
Five countries essentially telling CRU: "do NOT hand out our data to others".
[Ah, thank you, I've added that to the "update" -W]
Posted by: Marco | March 8, 2010 1:19 AM
Thanks for the link. But I wasn't the first to notice that the transcript say it was Acton that was talking about Sweden. I found the link somewhere in the comments section of a blog and decided to fact-check.
Posted by: Max Andersson | March 8, 2010 7:36 AM
Bill's comments above make no sense to me.
If Dr. Jones wasn't bound by restrictions from SMHI, why wouldn't he have released the data as he did in many other cases where there was no restriction? Why would he have bothered to ask permission needlessly?
And why would SMHI have replied--over-harshly or not--in the negative, as they did on December 21?
Posted by: Kevin McKinney | March 8, 2010 1:22 PM
While alarmists and skeptics parse personalities, the core issue remains. Is climate a good basis for or measurement of policy? I think not. My argument is not about the science in general or any of its particulars. After billions in research and advocacy and decades of political activity, we stand at a point of failure of the international process, weakening of public support, scientists thrown into irrelevant media scandal and constant nonproductive contention.
The desire to replace fossil fuels predates concern about climate. The case for replacing fossil fuels is stronger if climate is never mentioned.
Posted by: Paul Kelly | March 8, 2010 2:56 PM
40 Paul Kelly
"After billions in research and advocacy and decades of political activity, we stand at a point of failure of the international process, weakening of public support, scientists thrown into irrelevant media scandal and constant nonproductive contention."
All well and good, but there are people like this guy who see it as their destiny to make sure all of the above occurs.
Posted by: J Bowers | March 8, 2010 5:42 PM
Hmmm...
it is *not* true that Dr. Jones asserted that the weather services of several countries, including Sweden, Canada and Poland, had refused to allow their data to be released
From the transcript (after Acton makes his remark quoted above):
Professor Jones: There are constraints on some of the stations.
Q112 Mr Boswell: Withholding information, that is the other factor is it?
Professor Jones: Yes. We have tried to go back to the countries and seven countries have said they would rather we did not release the copies of their data we have in our database.
Q113 Mr Boswell: Do they give any indication why? I realise that is sensitive perhaps.
Professor Jones: It is not that sensitive. Canada, for example, says they would rather we sent requests for Canadian data to their website; they do not want us to put their data on our website.
Actually SMHI said that they didn't want Jones to release their data because Jones had changed it so much that it didn't look like their data anymore. Obviously, they didn't want to be held responsible for what Jones had done.
[Err, no, they didn't. Why do you feel this constant need to twist what they said? Why not just quote what they said instead of an inaccurate inflation? -W]
Posted by: bill | March 8, 2010 9:17 PM
Perhaps the lesson is to instead just read the transcript yourself.
Posted by: bill | March 8, 2010 9:30 PM
http://www.google.com/search?q=how+to+use+quotation+marks+correctly
Posted by: Hank Roberts | March 8, 2010 9:40 PM
Bill dear stop trying to twist things, it makes it clear that you are not someone worth trusting or talking to, although you certainly should be talked to.
Jones takes raw data sets and homogenizes them as has to be done in order to put stations and national raw data on a common basis to generate a consistent global data set. As the Swedes said, they also homogenize their data, but they do it differently than Jones.
Posted by: Eli Rabett | March 9, 2010 2:42 AM
The odd thing is, Eli, that I don't really have to twist things at all. Jones has more than twisted them enough. From what we learn after the fact, SMHI really doesn't have a problem with Jones publishing the data, as long as he includes a note that he has changed it. One can only wonder what their response would have been had he not lost it.
Oh, but did he lose it? From the transcript (Can there be anyone who hasn't read the transcript by now?):
"Lord Lawson of Blaby: Also, they have said that they have lost some of the data, so they cannot make it available. May I read what used to be on their website? It has been taken off their website, but this was on their website for a very long time until recently: "Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added, i.e. quality controlled and homogenised data." Some of the data they were saying at that time was lost. They are now saying it was not lost."
If there was no way for them to maintain the original, I guess I can't second guess that. But is that what happened? Did they lose it or not? Presumably they did, or Jones would have had the originals. Can anyone imagine a scientist not keeping raw data, in the off chance that they have processed it incorrectly and need to process it again? I don't think even Jones knows what happened anymore.
[The mistake you're making is confusing raw data with a copy thereof. To lose the original met records would be bad (though in the pre-huge-disk era I wouldn't be too surprised if people had) but that isn't the issue. The raw data belongs to the people that created it, which is the national met services. This point has been made again and again, so I'm disappointed you haven't understood it yet -W]
Were they longing to publish it as Acton says, or were they refusing data to Hughes for abhorrent reasons and sending emails saying they would rather delete data than share it and sending emails asking how they could avoid FOI requests? Did they have restrictive agreements or not? Jones thinks that at one time he said that they had 5 of them.
[Yes, they had restrictive agreements, and you've seen them, so why are you asking?
Irrelevance redacted, since I'm here -W]
Posted by: bill | March 9, 2010 7:49 PM
46 bill: "If there was no way for them to maintain the original, I guess I can't second guess that. But is that what happened? Did they lose it or not? Presumably they did, or Jones would have had the originals. Can anyone imagine a scientist not keeping raw data, in the off chance that they have processed it incorrectly and need to process it again? I don't think even Jones knows what happened anymore."
Data need only be kept for 5 years after publication, and even then it's up to the policy of the institution unless federal funding is involved. If others are too slow in trying to debunk the work, that's their problem - they need to get out of bed earlier.
Licensed data has to be deleted as soon as the work is done, according to agreements.
http://www.cru.uea.ac.uk/cru/data/availability/agreements.pdf
Posted by: J Bowers | March 9, 2010 8:01 PM
Link for the above posted guidelines:
http://ori.dhhs.gov/education/products/clinicaltools/data.pdf
Posted by: J Bowers | March 9, 2010 8:06 PM
So it has all been deleted, then? Or only some of it? Do they know who requires deletion? Did they delete any of the data from the 52 who said they could publish it? That would be odd, wouldn't it. Or does this mean that they were remiss, and they didn't delete data that they should have deleted and you are now unearthing evidence that they are committing a crime? These are actually not rhetorical questions. If you know the answer, please feel free to reply.
Posted by: bill | March 9, 2010 8:10 PM
@ bill
I hope it's all been deleted, even though it hasn't. Were up to me, I'd delete it all just for the hell of it. I'd video the event and post it on Youtube, one finger hovering over the 'delete' key and another stuck up at the camera.
Otherwise, you could always just follow the links and read them before your concentration span expires. As for the more intimate details of what exactly was deleted when and where and by whom, why don't you break the vicious circle you and your chumpy chums are in and stop expecting everything to be handed to you on a plate.
Posted by: J Bowers | March 9, 2010 8:22 PM
Wow, clever. Chumpy chums. Nice. Good jab about my limited concentration, too.
It is an interesting document. Does it have anything to do with Jones? Your statement is so ridiculous, I'm not exactly sure how to reply. I'm not expecting Jones to hand me anything, especially not on a plate. Except the truth. You can make a smarmy reply to me, but that is because you have no idea what Jones has been doing, either. No one does. That is the point. Not even Jones knows what he has been doing. He thinks he said that he had 5 countries that they had data agreements. That's almost comical. Where in your document about handling data does it say anything about almost keeping track of your data and where it came from? But there I go, asking for something on a plate again...
Posted by: bill | March 9, 2010 10:18 PM
Why would you keep a duplicate set of data when you could go back to the NMOs and get whatever you needed if you needed it? J Bowers is right, the clowns are going to make sure that everything is erased for self protection. Another thing that is going to happen is FOI requests are only going to be answered if costs are covered.
Posted by: Eli Rabett | March 9, 2010 11:13 PM
"Why would you keep a duplicate set of data when you could go back to the NMOs and get whatever you needed if you needed it?"
I guess as long as you were 100% sure that they would always be there to give you the data and that they would give you the data with the same usage restrictions and you were certain that they wouldn't change the way they homogenized the data (possibly invalidating your own routines) and that it wouldn't cost you money in the future to get something you already have and you wouldn't want to go through what might be a hassle to get the data again... No, I can't think of any reasons to just hold on to the data.
Posted by: bill | March 9, 2010 11:50 PM
51 bill: "He thinks he said that he had 5 countries that they had data agreements. That's almost comical."
Et tu quoque. What's really comical is self-styled "policemen of science" who make serious allegations then get to court only to find they can't figure out what they wrote in their own notebooks. It only happens to be at the core of their case against Briffa. Has McI stopped being so busy so he can look into it, yet?
http://www.realclimate.org/?comments_popup=3041#comment-163936
http://shewonk.wordpress.com/2010/02/26/the-inquiry-submissions/#comment-1625
53 bill: "I guess as long as you were 100% sure that they would always be there to give you the data and that they would give you the data with the same usage restrictions and you were certain that they wouldn't change the way they homogenized the data (possibly invalidating your own routines) and that it wouldn't cost you money in the future to get something you already have and you wouldn't want to go through what might be a hassle to get the data again..."
Sorry to bring up inconvenient truths concerning legally binding agreements.
http://www.cru.uea.ac.uk/cru/data/availability/agreements.pdfPosted by: J Bowers | March 10, 2010 6:07 AM
Can anyone tell me how many national meteorological offices (those holders of those nations' meteorological data) have disappeared in the last, say, 100 years and whose data has been subsequently lost because of that closure (noting that some local data collections do sometimes go missing, at least temporarily)?
Can anyone please explain to me how a change in homogenisation procedures could lead to "possibly invalidating your own routines". Isn't the output of the homogenisation procedure a set of numbers that are in essence only slightly different in some of the magnitudes from the raw data set?
And can anyone please explain to me why, having had homogenisation changes (and possibly transcription errors corrected and new data added too) one would want the old original data set rather than the new original data set at some later date to derive a third-party product?
[There are genuine difficulties here.
If you were doing this now, I think you would carefully archive the original dataset (or its digitisation) on a tiny corner of your smallest disk, or on the memory stick you got with your cornflakes. But anyone (and I'm sure this includes my readers) familiar with disk technologies from the 1980's will recall that saving everything was not possible. So while it would be nice to have the original, plus a carefully applied set of diffs recording changes, this simply isn't realistic in the real world.
Real people know this. Asking otherwise is like those who ask for "total openness" is afaux-naive way: no-one wants that, no-one applies it to themselves, but it makes a good sound bite -W]
Posted by: P. Lewis | March 10, 2010 7:19 AM
55 P Lewis: "And can anyone please explain to me why, having had homogenisation changes (and possibly transcription errors corrected and new data added too) one would want the old original data set rather than the new original data set at some later date to derive a third-party product?"
One explanation is if you're looking to discredit the science, not by doing actual science which you don't understand in the first place. Couple that with an inability to differentiate between an honest mistake and fraud, politicians and groups trying to influence public policy with dubious agendas, plus a pack of admirers susceptible to dog-whistles, then you have the makings of a tragicomedy.
Posted by: J Bowers | March 10, 2010 9:07 AM
"But anyone (and I'm sure this includes my readers) familiar with disk technologies from the 1980's will recall that saving everything was not possible."
You should have seen the data store at the UKMO in the late eighties. The effort made to convert the data from bespoke flat file format to relational databases in the early nineties was a reasonable effort, too.
And that's for a poxy country the size of the UK (though there were lots of buoy and ship obs too). ;)
The amusing thing though, was the complaints from the people (internal to the UKMO) who used to get the data in the form of micro-fiche and wanted that to remain the case - despite the fact that they were (going to be) given direct read access to the (entire?) dataset.
Posted by: Adam | March 10, 2010 9:12 AM
William, I hope you realise I was commenting on bill's #53 comment.
[I'm losing track.. :-) -W]
I'm acutely aware of the data archiving problems. I have floppy disks now with no machine to access them; and I similarly have zip disks with no means to access them. Though, at the moment, it would not be onerous to locate and fit the necessary drives should I wish to access the data and routines on them.
Of course, third parties archiving raw data in the manner you sugggest would presumably, then and now, be transgressing the set licence conditions on the handling of that data (more so "then", presumably, since there seems latterly to be a move to more willingness by NMOs for their raw data to be disseminated more widely).
On the issue of raw data contents at various time points, I've been presuming that the NMOs could, if they wished, supply or reconstruct the data for a particular time point. Is that the case?
[I think there would be doubt about that, though I'm not familiar enough with NMO operation to be sure. Imagine, for example, that soeone is digitising the original records, by hand. They come across an obvious transcription error: two digits that have been transposed, say, that is either obvious from preceeding obs, or becomes obvious when looking at a nearby station. Will they make a careful note of that (will there even be a place for it?) or will they just correct the digitised version? In which case, whilst in theory someone could go back and re-check, in practice no-one will ever have the time -W]
Posted by: P. Lewis | March 10, 2010 9:24 AM
Oh, back in the day, archivers would keep 9-track magnetic tape in climate controlled storage at great expense. Didn't help when it turned out that one manufacturer's magnetic coating chemistry degraded over some years even in such conditions, leading to it delaminating and flaking. This was IIRC back in the 80s or perhaps the early 90s. I think it might've affected the music industry, too, i.e. the loss of archived analog masters.
I've pointed out elsewhere (and perhaps here) that when CRU lost some data in an office move in the mid-80s, the movers probably lost a box of 9-track tapes. People today just don't relate to just how physically massive magnetic media (disks as well) were back in the 1980s.
Posted by: dhogaza | March 10, 2010 10:12 AM
The UKMO (used to have?) a room-sized robot that switched tapes in the 80s/90s. IIRC you (sometimes?) had to request a tape to be loaded for some overnight runs...but that's a bit murky now. There may have been an entry in the job control header to request it.
[Yup, I remember that. I had to ask it for radiosonde data, though I think that was on even oldre tapes. Not all of them were readable... -W]
Posted by: Adam | March 10, 2010 10:25 AM
Wow. Michaels blames _Jones_ for causing the editors of Climate Research to 'resign or be fired'
http://www.cato.org/pub_display.php?pub_id=11022
He also claims he has a list of .... oh, why bother.
I wonder if there's a Darwin Award at the species level.
I suspect we're going to get it.
So to speak.
Posted by: Hank Roberts | March 10, 2010 11:29 AM
AFAIA no data was lost before the switch to the MIDAS system, but IIRC they very nearly lost all the rainfall data in a hardware upgrade - there was some very clever memory saving programming going on in the earlier databases (written in the '70s) that relied on some OS foibles. It survived several upgrades (including a manufacturer change) but was not going to survive any more - hence the DB migration.
Anyway, not long after the initial project finished, there was another overhaul needed: http://www.trustededge.com/docs/ukmet.pdf ( - apologies it's a commercial case study).
Posted by: Adam | March 10, 2010 11:47 AM