Climategate - it's NOT just the emails

[Keepitsimple]

From Lorrie Goldstein's column this morning. As an old programmer, I know very well that projects fail because developers did not understand the data and the requirements; precisely what information they had - and what they wanted to do with it. Lorrie's article, which describes one such critical CRU project - demonstrates the Charley Chaplinesque attempts by a CRU developer to "model" temperatures. It's simply mind-boggling......and it's only the tip of the iceberg:

'Botch after botch after botch'

Leaked 'climategate' documents show huge flaws in the backbone of climate change science

By LORRIE GOLDSTEIN

I've been poring over one of many leaked computer files from the "climategate" scandal.

It's worse than those e-mails revealing leading climate scientists did a "trick" to "hide the decline" in global temperatures and privately called it a "travesty" they couldn't explain recent cooling.

This document has the innocuous header "HARRY_READ_Me.txt."

I'm indebted to Kate McMillan, the remarkable Canadian blogger who runs smalldeadanimals.com, for calling it to my attention.

You can easily find it online. I used www.anenglishmanscastle.com/HARRY_READ_Me.txt.

The file -- 274 pages long -- describes the efforts of a climatologist/programmer at the Climatic Research Unit (CRU) of the University of East Anglia to update a huge statistical database (11,000 files) of important climate data between 2006 and 2009.

The computer coding, along with the programmer's apparently unsuccessful efforts to complete the project, involve data that are the foundation of the study of climate change -- recordings from hundreds of weather stations around the world of temperature and precipitation measurements from 1901 to 2006, sun/cloud computer simulations, and the like.

As you read the programmer's comments below, remember, this is only a fraction of what he says.

- "But what are all those monthly files? DON'T KNOW, UNDOCUMENTED. Wherever I look, there are data files, no info about what they are other than their names. And that's useless ..." (Page 17)

- "It's botch after botch after botch." (18)

- "The biggest immediate problem was the loss of an hour's edits to the program, when the network died ... no explanation from anyone, I hope it's not a return to last year's troubles ... This surely is the worst project I've ever attempted. Eeeek." (31)

- "Oh, GOD, if I could start this project again and actually argue the case for junking the inherited program suite." (37)

- "... this should all have been rewritten from scratch a year ago!" (45)

- "Am I the first person to attempt to get the CRU databases in working order?!!" (47)

- "As far as I can see, this renders the (weather) station counts totally meaningless." (57)

- "COBAR AIRPORT AWS (data from an Australian weather station) cannot start in 1962, it didn't open until 1993!" (71)

- "What the hell is supposed to happen here? Oh yeah -- there is no 'supposed,' I can make it up. So I have : - )" (98)

- "You can't imagine what this has cost me -- to actually allow the operator to assign false WMO (World Meteorological Organization) codes!! But what else is there in such situations? Especially when dealing with a 'Master' database of dubious provenance ..." (98)

- "So with a somewhat cynical shrug, I added the nuclear option -- to match every WMO possible, and turn the rest into new stations ... In other words what CRU usually do. It will allow bad databases to pass unnoticed, and good databases to become bad ..." (98-9)

- "OH F--- THIS. It's Sunday evening, I've worked all weekend, and just when I thought it was done, I'm hitting yet another problem that's based on the hopeless state of our databases." (241).

- "This whole project is SUCH A MESS ..." (266)

And based on stuff like this, politicians are going to blow up our economy and lower our standard of living to "fix" the climate?

Are they insane?

Link: http://www.torontosun.com/comment/columnists/lorrie_goldstein/2009/11/29/11967916-sun.html

Edited November 29, 2009 by Keepitsimple

[Michael Hardner]

And based on stuff like this, politicians are going to blow up our economy and lower our standard of living to "fix" the climate?

Really, it makes as much sense as writing off AGW based on some incomplete project notes from a programmer.

[Riverwind]

Really, it makes as much sense as writing off AGW based on some incomplete project notes from a programmer.

The FDA and other government regulatory bodies have very strict rules about procedures and software developed during testing of a drug. If these guidelines are not followed the drug cannot get approval becauses the results cannot be trusted. The fact that the results might be right anyways is irrelevant.

That is the issue here. We cannot trust results produced from such a codebase therefore we cannot act on them - even if they are true. We need to rebuild these temperature records from scratch.

[Michael Hardner]

The FDA and other government regulatory bodies have very strict rules about procedures and software developed during testing of a drug. If these guidelines are not followed the drug cannot get approval becauses the results cannot be trusted. The fact that the results might be right anyways is irrelevant.

That is the issue here. We cannot trust results produced from such a codebase therefore we cannot act on them - even if they are true. We need to rebuild these temperature records from scratch.

Well, I can't understand how a study can be published without making the data available for scrutiny anyways, as well as the models. Any problems they had developing the paper is beside the point, as long as the data and processes are transparent.

[Riverwind]

Well, I can't understand how a study can be published without making the data available for scrutiny anyways, as well as the models.

The reason is simple: the media allowed themselves to be duped into believing that sceptics of AGW had nothing important to say and refused to cover the issues they raised. We see that trend continuing in the MSM where almost no outlet is covering the Climategate issue outside of opinion columns.

Any problems they had developing the paper is beside the point, as long as the data and processes are transparent.

The problems are important when scientists ignore problems with processes because the process produces results they want to believe. Sites like climate audit have documented numerous examples of these failures.

That is why the governments need to step in an make it clear that science that does not follow proper processes is not to be used in any government policy making.

Edited November 29, 2009 by Riverwind

[Michael Hardner]

The reason is simple: the media allowed themselves to be duped into believing that sceptics of AGW had nothing important to say and refused to cover the issues they raised. We see that trend continuing in the MSM where almost no outlet is covering the Climategate issue outside of opinion columns.

Is data review normally part of peer review or not ? If so, when did that change with GW studies and what was the reason that was given for the change ?

[Michael Hardner]

double post

Edited November 29, 2009 by Michael Hardner

[Riverwind]

Is data review normally part of peer review or not ? If so, when did that change with GW studies and what was the reason that was given for the change ?

Peer reviewers almost never check the data. All they really do is look at the presentation in the paper to see if the conclusions are supported by the presentation. The data could be completely fabricated and the peer reviewer would never know.

All scientists know this but that has not stopped alarmists from selling peer review as an infalliable quality control mechanism when nothing could be further from the truth.

This is a point that SteveMc has been trying to make for years but everyone ignores him because he is labelled as a 'denier'.

The fact is our scientific processes are completely broken and they need reform and this is not going to happen until our political leaders have the guts to confront the scientists and inform them that their attitudes are no longer acceptable if they wish to participate in science that is used to develop public policies.

Edited November 29, 2009 by Riverwind

[Riverwind]

Here is the latest example of dishonest science coming out of CRU.

Even in their Nov 24, 2009 statement, the University of East Anglia failed to come clean about the amount of decline that was hidden. The graphic in their statement continued to “hide the decline” in the Briffa reconstruction by deleting adverse results in the last part of the 20th century. This is what Gavin Schmidt characterizes as a “good thing to do”.

If you follow the link you will see that UEA published a graph which still leaves out a chunk of data which undermines the case they wanted to make. The fact that this graph was published after the scandal broke gives ua a lot of insight into the duplicitous nature of these scientists and why they cannot be trusted.

[Michael Hardner]

Peer reviewers almost never check the data. All they really do is look at the presentation in the paper to see if the conclusions are supported by the presentation. The data could be completely fabricated and the peer reviewer would never know.

Why don't they check the data ? Aren't there scientists that are skeptical of GW that would be interested in looking at the data ?

All scientists know this but that has not stopped alarmists from selling peer review as an infalliable quality control mechanism when nothing could be further from the truth.

This is a point that SteveMc has been trying to make for years but everyone ignores him because he is labelled as a 'denier'.

Why doesn't he just get somebody with credentials to review the data ?

The fact is our scientific processes are completely broken and they need reform and this is not going to happen until our political leaders have the guts to confront the scientists and inform them that their attitudes are no longer acceptable if they wish to participate in science that is used to develop public policies.

You can't legislate good sense, as has often been said. If scientists are getting lazy in their thinking, I do trust them to pay attention and fix the process. It doesn't make sense to get government involved to solve every problem that's out there.

[Riverwind]

Why don't they check the data ? Aren't there scientists that are skeptical of GW that would be interested in looking at the data ?

Being a peer reviewer is a volunteer position. Checking data takes a lot of time. Sceptical scientists are under pressure to produce papers with 'new' results. Simply checking the data from another paper does not give any ROI. That is why SteveMc makes a distinction between auditing and science. Auditing only checks the numbers and does not attempt to develop new ideas or theories. It is a very important job that is not getting done and the garbage CRU code is the consequence.

Why doesn't he just get somebody with credentials to review the data ?

Who? SteveMc? He is more than qualified given his background in statistics and is willing to do it himself but he keeps being denied access to the data by the scientists.

I do trust them to pay attention and fix the process.

Please provide me some concrete examples of why climate scientists can be trusted to fix the process. I can give you a long list of examples that show they cannot be trusted - included the post I just made above.

I really urge you to reconsider your unquestioning trust in scientists because I feel it is not something that can be supported by a rational examination of the available evidence.

Edited November 30, 2009 by Riverwind

[Michael Hardner]

Being a peer reviewer is a volunteer position. Checking data takes a lot of time. Sceptical scientists are under pressure to produce papers with 'new' results. Simply checking the data from another paper does not give any ROI. That is why SteveMc makes a distinction between auditing and science. Auditing only checks the numbers and does not attempt to develop new ideas or theories. It is a very important job that is not getting done and the garbage CRU code is the consequence.

All scientists are under pressure to review papers, and although checking data can take a lot of time it doesn't need to.

Sorry, but it sounds like a cop-out to say that tons of data is being faked, or is inaccurate and nobody has looked, and nobody has time to look. That being said, it might be true. The banking scandal in the US in 2008 was a result of people who were paid to audit and review not doing so.

Who? SteveMc? He is more than qualified given his background in statistics and is willing to do it himself but he keeps being denied access to the data by the scientists.

I think you had this disagreement with Waldo earlier on the thread. I don't understand why he is being denied access while others are not ? Or does nobody have access ? This isn't making sense to me because earlier you said that scientists don't have time to check data so which is it ?

Please provide me some concrete examples of why climate scientists can be trusted to fix the process. I can give you a long list of examples that show they cannot be trusted - included the post I just made above.

I can't say for sure, however: It's an open process, where the problems that have been highlighted are clear to all so it should be evident to all whether the fixes for those same problems (whatever they may be) are working moving forward.

You gave some examples where the system seems to have failed. Those will have to be looked into to determine what happened, and whether anything needs to be fixed or not.

[waldo]

From Lorrie Goldstein's column this morning. As an old programmer, I know very well that projects fail because developers did not understand the data and the requirements; precisely what information they had - and what they wanted to do with it. Lorrie's article, which describes one such critical CRU project - demonstrates the Charley Chaplinesque attempts by a CRU developer to "model" temperatures. It's simply mind-boggling......and it's only the tip of the iceberg:

worthy of it's own thread, hey! C'mon, Simple... this one's been beat upon so heavily in the last days. But it's always refreshing to hear someone issue a supposed from authority revelation... coming from a self-described old programmer who, apparently, knows nothing how real software development does/may occur.

This isn't code... this is simply a text file of running commentary used by an individual tasked with coming in after-the-fact and working to update product documentation. Invariably, programmers are loath to create extensive documentation - this is not unique to any industry, to any science. More progressive development methodologies (e.g. expert agile methodology), in fact, view creating detailed static documentation as an impediment, as a detriment, to eventual success of the design, of the development. This approach has worked well for development shops; however, once production deployed, Operations Support may face challenges depending on the hand-off between Development and Operations. Invariably, Operations Depts... those left to manage the production ready code received from Development, have their own standards, inclusive of static documentation. Building that static documentation, after the fact, can be most intensive, most challenging, most frustrating and may reflect upon the skills, knowledge and... personality... of those tasked with writing (updating) the documentation. Some of that skill, knowledge and... personality... of the individual ('Harry') comes through in his self-documenting text file that reflects his after-the-fact efforts to update documentation.

... for completeness, this single individuals documentation update initiative reflects back upon a dated early legacy product, the CRU TS 2.1 product which has no relationship to the most current product associated with the CRU HadCRUT temperature dataset.

[Riverwind]

Sorry, but it sounds like a cop-out to say that tons of data is being faked, or is inaccurate and nobody has looked, and nobody has time to look. That being said, it might be true. The banking scandal in the US in 2008 was a result of people who were paid to audit and review not doing so.

Well it *is* true in climate science. Sceptics have know this for years but these emails are the first time that we have had independent evidence. The sooner people accept that there is a high possibility that such a failure has occurred the sooner we get move on to fixing it before it does more damage.

I don't understand why he is being denied access while others are not?

SteveMc is denied access because scientists don't want to give the data to someone who will make them look bad. They have no problems given the same data to their collegues because their collegues will not question it.

This isn't making sense to me because earlier you said that scientists don't have time to check data so which is it ?

It really depends on the data. Some data is available. Some data can be accessed if you ask the author. Many peer reviewers will simply not bother to look at the data even if is available.

You gave some examples where the system seems to have failed. Those will have to be looked into to determine what happened, and whether anything needs to be fixed or not.

Looked into by who? You seem to be arguing that the Liberal party should have been allowed to look into the sponsership allegations and determine if anything needs to be fixed. The conflict of interest makes such a suggestion absurd and the same conflict of interest exists for scientists and the instititions that depend on them to attract funding.

[Riverwind]

for completeness, this single individuals documentation update initiative reflects back upon a dated early legacy product, the CRU TS 2.1 product which has no relationship to the most current product associated with the CRU HadCRUT temperature dataset.

Another lie courtesy of RC. The code produces HadCRUTv3 as well as HadCRUTv2. More importantly the code shows that HadCRUTv3 was adjusted by fitting a curve to the HadCRUTv2 results which means any errors in HadCRUTv2 will be in HadCRUTv3. Edited November 30, 2009 by Riverwind

[Michael Hardner]

Well it *is* true in climate science. Sceptics have know this for years but these emails are the first time that we have had independent evidence. The sooner people accept that there is a high possibility that such a failure has occurred the sooner we get move on to fixing it before it does more damage.

SteveMc is denied access because scientists don't want to give the data to someone who will make them look bad. They have no problems given the same data to their collegues because their collegues will not question it.

I don't want to hear your suspicions as to why he is not given the access while others are, I would like to hear the stated reason so that I can decide for myself thanks.

It really depends on the data. Some data is available. Some data can be accessed if you ask the author. Many peer reviewers will simply not bother to look at the data even if is available.

Well all of these are different problems.

1) Data should be available.

2) Skeptics should ask the author for the data.

3) Peer reviewers should look at the data.

None of this sounds conspiratorial, but there may be some neglect happening.

Looked into by who? You seem to be arguing that the Liberal party should have been allowed to look into the sponsership allegations and determine if anything needs to be fixed. The conflict of interest makes such a suggestion absurd and the same conflict of interest exists for scientists and the instititions that depend on them to attract funding.

The Sponsorship allegations are not a good analogy. If I remember correctly, money was secretly siphoned off to organizations that supported the liberal party. These papers are published and reviewable in an open way. If, for example, the lack of data turns out to be a problem then the correction should be to provide the data with publishing, and the audit will be easy: did they provide the data ?

The fact that all of this is being done in the open doesn't help support the view that this is a conspiracy. The idea that it takes a lot of time to check data doesn't support that view either, as there are ways to check the data through random sampling.

We're now more than 1 week into this, and we haven't seen anything yet that supports a wide conspiracy.

[Michael Hardner]

Wait... news flash...

The Telegraph

In a statement welcomed by climate change sceptics, the university said it would make all the data accessible as soon as possible, once its Climatic Research Unit (CRU) had negotiated its release from a range of non-publication agreements.

The publication will be carried out in collaboration with the Met Office Hadley Centre. The full data, when disclosed, is certain to be scrutinised by both sides in the fierce debate.

That sounds like what I prescribed above...

[waldo]

Another lie courtesy of RC. The code produces HadCRUTv3 as well as HadCRUTv2. More importantly the code shows that HadCRUTv3 was adjusted by fitting a curve to the HadCRUTv2 results which means any errors in HadCRUTv2 will be in HadCRUTv3.

well... we see the 'lie' word again emanating from the mouths of the deniers

pony up your source for that, bucky!

[Riverwind]

I don't want to hear your suspicions as to why he is not given the access while others are, I would like to hear the stated reason so that I can decide for myself thanks.

Their reasons are stated clearly in the leaked emails. They don't want SteveMc to have the data because he would find things wrong with it ans they would have 'waste time' defending themselves.

Well all of these are different problems.

1) Data should be available.

2) Skeptics should ask the author for the data.

3) Peer reviewers should look at the data.

The problem is

1) Key datasets are not available.

2) Authors refuse to provide data to sceptics

3) Peer reviewers don't look at the data.

None of this sounds conspiratorial, but there may be some neglect happening.

It is not conspiracy or neglect. It is a system with no reliable quality control mechanism that is producing suspect results.

The fact that all of this is being done in the open doesn't help support the view that this is a conspiracy. The idea that it takes a lot of time to check data doesn't support that view either, as there are ways to check the data through random sampling.

This stuff was not being done in the open. The data and code was hidden from people that might criticize it. And when the data and code is made available the climate scientist establishment spends its time attacking the credibility of the critics instead of actually fixing problems. Edited November 30, 2009 by Riverwind

[Riverwind]

pony up your source for that, bucky!

You know you could avoid embarrasing yourself if you took time to fact check the pronouncements from RC. You will find that honesty is not their strong suit.

From the README file for the code that produces the HadCRUT temperature series.

I took the twelve 1990 anomaly files from the original 1901-2006 run (that was done with

some flavour of anomdtb.f90). They were here:

/cru/cruts/version_3_0/primaries/tmp/tmptxt/*1990*

Then I modified the update 'latest databases' file to say that tmp.0705101334.dtb was the

current database, and made a limited run of the update program for tmp only, killing it

once it had produced the anomaly files. The run was #0908181048.

So, under /cru/cruts/version_3_0/fixing_tmp_and_pre/custom_anom_comparisons, we have a

'manual' directory and an 'automatic' directory, each with twelve 1990 anomaly files. And

how do they compare? NOT AT ALL!!!!!!!!!

Example from January:

crua6[/cru/cruts/version_3_0/fixing_tmp_and_pre/custom_anom_comparisons] head manual/tmp.1990.01.txt

For some reason I don't think they would store v2 data in directories labeled version_3_0

Edited November 30, 2009 by Riverwind

[waldo]

You know you could avoid embarrasing yourself if you took time to fact check the pronouncements from RC. You will find that honesty is not their strong suit.

From the README file for the code that produces the HadCRUT temperature series.

For some reason I don't think they would store v2 data in directories labeled version_3_0

c'mon bucky... name your source... don't be hesitant. You obviously know nothing of software development... name the source you parrot from.

[Riverwind]

c'mon bucky... name your source... don't be hesitant. You obviously know nothing of software development... name the source you parrot from.

The README from the HadCRUT code I have on my computer. You can find and download the code yourself. Edited November 30, 2009 by Riverwind

[Riverwind]

That sounds like what I prescribed above...

It still does not address the culture of entitlement that allowed CRU to keep its data and code secret for so long. The claims of the climate science establishment simply cannot be trusted until they publically accept that they were wrong, apologize and demonstrate a willingness to reform the processes they use to ensure the integrity of their results. Edited November 30, 2009 by Riverwind

[Michael Hardner]

It still does not address the culture of entitlement that allowed CRU to keep its data and code secret for so long. The claims of the climate science establishment simply cannot be trusted until they publically accept that they were wrong, apologize and demonstrate a willingness to reform the processes they use to ensure the integrity of their results.

You may indeed get a correction and a statement from the university, and from the professors.

Riverwind - what's this about keeping its data secret ? You already said that it was available but it took to long to analyze.

[Riverwind]

Riverwind - what's this about keeping its data secret ? You already said that it was available but it took to long to analyze.

Look there are a lot of different datasets used in many different situations. Some of this data was made publicly available or was given to sceptics who asked politely for it. However, significant datasets like the GISS and HadCRUT datasets and methods were keep secret until some sort of scandal broke that brought the politicians in who forced the agencies to make the data and methods public.

It should not require a scandal to get access to data and the scientists who felt they were entitled to keep the data secret should be held to account. An admission of wrong doing and a public apology would be a start but there needs to been an end to this case by case battle for data. Politicians need to step in and make it clear that such secrecy is not acceptable. If they do not do it then the same problem will repeat over and over again with difference agencies and scientists.

Edited November 30, 2009 by Riverwind