IPCC Rejects Anonymous Review

Although the IPCC calendar webpage doesn’t link to session documents of the 34th session (Kampala Nov 2011), David Holland has alertly located the documents – see here.

IPCC rejected a proposal for anonymous peer review – see document here (page 12 on).

We haven’t discussed this topic previously (in an IPCC context). On reflection, the adoption of a form of anonymous peer review by IPCC seems to me to be a very good idea and might somewhat mitigate some problems. There is no doubt in my mind that review responses are strongly conditioned by who is making the suggestion. The experiences of Ross and myself are vivid examples.

In the IPCC’s Special report on Carbon Dioxide Capture and Storage of WG III of the Fourth Assessment Report, reviews were anonymous. The review document described the results of anonymous review as being entirely positive:

The TSU prepared a list of Review comments with numbers. The TSU had a conversion table linking the numbers to the Reviewer’s names. During the treatment at the lead author meetings and the formulation of the authors’ responses the authors only saw the numbers. The authors knew that in case they would need to consult an Expert Reviewer for getting some clarifications about his comments, the anonymity could be lifted and the coordinates of the Expert Reviewer would have been made available to the authors. In practice it turned out they did not need to use this provision. The anonymity was continued until finalization of the final draft report.

The Reviewers and authors have been informed beforehand about this procedure. The number of comments was normal compared to other special reports. No Reviewer used improper or inappropriate language. The WG III co chairs and TSU held an enquiry among the authors and Review Editors. They considered the anonymity an improvement, because it made them concentrate fully on the content of the matter, disregarding the persons and their background, which was more time efficient.

The review document summarized the advantages of anonymous review as follows:

• Authors will concentrate on the content of the matter, excluding (subconscious) biases.
• There is positive experience in WG III AR4 – also the Task Force on the Greenhouse Gas Inventory Program (TFI) has practiced anonymous reviews with a positive judgment of the authors and Review Editors.
• Authors cannot be criticized anymore of ignoring comments of specific individuals or representatives of scientific schools or interest groups, as happened in the past.
• It remains possible for authors to contact Expert Reviewers if there is a need for clarification.

These seem pretty convincing arguments. Here are the arguments against:

• The Task Group on Procedures was installed in order to consider the recommendations of the InterAcademy Council (IAC). The IAC did not recommend anonymous Reviews, so there is no compelling reason to address this.
• There is a risk that Exper Reviewers could take advantage of their anonymity by burdening authors with unprofessional or inappropriate comments.
• Measures against biases are already sufficiently taken by having Review Editors and by having a authors working as a group.
• Transparency is crucial to the IPCC process. There would be an imbalance in transparency when authors are known by name and Reviewers are not.
• IPCC needs a consistent approach with regard to its Expert Reviews. Changing the approach to require anonymous Review comments would imply that there is a problem with the named Reviewer approach, which is not the case.
• Named Expert Review is more efficient as it allows writing teams to liaise with Reviewers when there is a need for clarification.

In my opinion, none of these reasons stands up.

The Task Group on Procedures was installed in order to consider the recommendations of the InterAcademy Council (IAC). The IAC did not recommend anonymous Reviews, so there is no compelling reason to address this.

The first argument starkly shows the hypocrisy and opportunism of IPCC. IAC did not recommend (or even consider) Jones-Stocker enhanced confidentiality. This was not not mentioned in the briefing documents for the Jones-Stocker amendment. Worse, they represented the changes as addressing issues raised by the IAC.

There is a risk that Exper Reviewers could take advantage of their anonymity by burdening authors with unprofessional or inappropriate comments.

The IPCC’s own experience with the Carbon Dioxide Task Group was reported to be the opposite. To the extent that accountability was an issue, the reviewer names could be removed from the version given to authors for comment, but restored in the final publication of review comments, thereby ensuring accountability.

Measures against biases are already sufficiently taken by having Review Editors and by having authors working as a group.

Neither of these measures had the slightest deterrent to AR4 author responses. In practice, authors seem to have divided up responsibilities in their chapter and to have been busy handling their own sections without worrying too much about how, for example, Briffa handled review comments in his section.

Transparency is crucial to the IPCC process. There would be an imbalance in transparency when authors are known by name and Reviewers are not.

I agree that transparency is “crucial” to the IPCC process. As discussed elsewhere, IPCC has opposed transparency in favour of confidentiality, with the situation getting worse with the furtive adoption of the Jones-Stocker amendment. In addition, it would be easy enough to add back the reviewer name when the review comments were published. The present system is designed not for transparency, but to enable authors to decide how to respond, depending on who the reviewer was.

IPCC needs a consistent approach with regard to its Expert Reviews. Changing the approach to require anonymous Review comments would imply that there is a problem with the named Reviewer approach, which is not the case.

This is perhaps the stupidest argument – even by IPCC standards. Once again, the pretence of infallibility. There are problems with the named reviewer approach. I can understand an argument that, after considering a balance of problems, an institution might choose one method rather than another. But worrying about the impact on infallibility is not a valid reason.

Named Expert Review is more efficient as it allows writing teams to liaise with Reviewers when there is a need for clarification.

Again, this is a fatuous argument. Their own experience with the Carbon Dioxide Capture Task Group permitted authors to locate reviewers for follow-up if necessary. In addition, there is little evidence from the Climategate emails that AR4 reviewers bothered to do this. Briffa, for example, didn’t try to clarify things with me or Ross.

Geoffrey Boulton and IPCC Secrecy

Phil Jones’ written answers to the Muir Russell panel shed interesting light on the insularity of IPCC authors, who see nothing odd about a system in which reviewers do not see either author responses to their review comments or the comments of other reviewers until long after the release of the final document. Jones’ comments were made in connection with questions from Boulton about Jones’ threat to keep McKitrick and Michaels 2004 out of IPCC and McKtrick’s allegation that Jones and other chapter 2 authors, having grudgingly agreed to refer to McKtrick and Michaels 2004, had “fabricated” IPCC’s editorial comment that its results had no statistical significance. This topic is revisited in AR5 (First Draft) where IPCC (with surprising candour) admitted that there was no “explicit” basis for the disparaging claim in AR4.

Another IPCC Demand for Secrecy

Unfortunately, IPCC seems far more concerned about secrecy than in requiring its contributors to archive data. I received another request to remove discussion of IPCC draft reports. On this issue, David Appell and I are in full agreement – see David Appell’s collection of ZOD chapters here. (Jan 30 Update - see below.)

More Spurious East Anglia Arguments Rejected

An excellent article at Bishop Hill here describing a clean sweep for Don Keiller in court (with David Holland as a “Mackenzie friend”) against the University of East Anglia and its solicitors. Decision is here.

The article reports on Keiller’s appeal to the First-Tier Tribunal (Case No. EA/2011/0152) in the General Regulatory Chamber – Information Rights. The appeal related to the second part of Keiller’s original FOI – instructions sent by CRU to Georgia Tech on their use of CRUTEM. CRU had argued that they didn’t have the information because Jones had deleted the email and they did not have access to the server in police possession. All arguments by the university were dismissed, with the judge being more than somewhat acid in some of his comments.

The University of East Anglia’s argument was hampered by their failure to present direct evidence from Phil Jones. (Assertions by Jones were presented by what Don Keiller described as “third-hand hearsay” – a conversation between Jones and David Palmer, passed on to Jonathan COlam-French, passed on to the UEA solicitor. Keiller and Montford observe:

It appears that UEA were keen that Jones should not appear on the witness stand, where he would be required to give evidence under oath. In fact it is noteworthy that, despite all the official “investigations”, Jones has never been required to answer questions under oath or provide a signed declaration.

Neukom and the Steig Over/Under

Earlier this year, I reported on the refusal of Raphael Neukom, an associate of IPCC confidentiality advocate and WG1 Co-Chair Thomas Stocker at the University of Bern, to archive data used in a then recent multiproxy study, Neukom et al 2011 (Clim Dyn). In his refusal letter, Neukom stated that

Most of the non-publicly available records were provided to us for use within the PAGES LOTRED-SA initiative only

Neukom’s website lists a set of multiproxy data, many of the series said to be “available on request for LOTRED-SA contributors”.

Despite Neukom’s “excuse” that the data had been provided to him for use “within the PAGES LOTRED-SA initiative only“, Neukom et al 2011 has been prominently used in the IPCC AR5 draft assessment report. The Neukom reconstruction is used to compare the performance of models to paleoclimate reconstruction in South America in FOD Figure 5.9 as shown below:

Figure 1. Neukom et al 2011 south South American temperature reconstruction (heavy black). Colored lines show results from various climate models.

Unless you are a climate scientist, you would probably not describe the paleoclimate reconstruction as cohering particularly well with the various models, but that’s a story for another day.

In the running text of the draft AR5, Neukom et al is cited on several occasions, including as authority for the observation that SH summer temperatures in the Medieval Warm Period (you know, the one that is supposedly regionally restricted to Greenland and a few counties in England) were “mostly warmer than the 20th century climatology”:

Progress has been made in the SH since AR4, where new tree ring records from the Andes, northern and southern Patagonia, Tierra del Fuego, New Zealand and Tasmania (Boninsegna et al., 2009; Cook et al., 2006; Villalba et al., 2009), ice cores, lake sediments and documentary evidence from southern South America (Neukom et al., 2011; Prieto and García Herrera, 2009; Tierney et al., 2010a; Vimeux et al., 2009; von Gunten et al., 2009) and terrestrial and shallow marine geological records from eastern Antarctica (Verleyen et al., 2011) allow a better understanding of past temperature variations (Neukom et al., 2011). A multi-proxy reconstruction for southern South American (Neukom et al., 2011) finds austral summer temperatures between 900 CE and 1350 CE that are mostly warmer than the 20th century climatology (though associated with large uncertainties), with a sharp transition after 1350 CE to colder conditions that last until approximately 1700 CE.

Citation of Neukom et al 2011 by IPCC clearly takes its use outside the realm of LOTRED-SA associates, but Neukom has thus far taken no steps to ensure that proxy data of Neukom et al 2011 (now used by IPCC) is available to anyone outside his circle of cronies.

Neukom (together with Joelle Gergis) has also published a recent survey of Southern Hemisphere proxies Neukom and Gergis 2011, one of the objectives of which was to report on the “availability” of SH proxies. Unfortunately, with the sort of blindness all too familiar from Climategate, Neukom and Gergis conflate availability of data to cronies with public availability – a situation that I am presently testing with a long-unarchived Eric Steig data set.

Neukom and Gergis 2011 is one of a number of recent multiproxy articles appearing more or less on the eve of AR5, as they themselves note:

Given the importance of global circulation features like ENSO, IOD, SAM and the IPO, a concerted effort is now underway to consolidate existing high-resolution palaeoclimate records from these regions in time for the IPCC fifth assessment report (AR5) (Gergis et al., 2011; Neukom et al., 2010, 2011).

They provide a list of 174 proxies (expanded from the list on Neukom’s website) that supposedly meet the following criteria:

- extend prior to 1900
- are calendar dated or have at least 70 age estimates in the 20th century
- extend beyond 1970 to allow sufficient overlap with instrumental records
- are accessible through public data bases or upon request from the original authors

Series said to be available “upon request from the original owners” are, for the most part, the same series that had previously been described as only “available on request for LOTRED-SA contributors”. Neukom and Gergis report that there were only 14 records extending back to 1000 AD that met the above criteria. Somewhat surprisingly, although IPCC had claimed “progress since AR4″ in the development of SH proxy data, only one of the 14 series appears to have been developed since AR4 (with most of the series being developed prior to AR3.)

Many of these records are listed by Neukom and Gergis as only “available upon request”. The most intriguing such example is Eric Steig’s Siple Dome dD (and d18O) series – see excerpt below – which Neukom obtained as a “personal communications”.

The Siple Dome core was drilled in 1993-4: the data was publicly funded. Even by Lonnie Thompson standards, this is a long time for the data to remain both unpublished and unarchived, particularly given the scarcity of long SH proxies. One cannot help but think that the data set would have been promptly published if it had HS shape and, ergo, my prediction is that, if and when, the data ever is made “available”, it will not have a Hockey Stick shape.

As a test of Neukom and Gergis’ statement that the data sets are “available upon request”, I submitted a request for this data to Eric Steig (cc Neukom) a couple of days ago, thus far, without acknowledgement from either.

Dear Dr Steig,
Neukom and Gergis 2011 stated that the datasets listed in their tables were accessible “upon request from the original authors”. Among their series are Siple Dome delD (1000-1993) and d18O (1654-1994) series said to have been sent to Neukom as a “pers comm.” In 2010. Could you please provide me with a copy of the series that you provided to Dr Neukom.

Thank you for your attention,
Stephen McIntyre

My early line on the over/under for getting the data is the publication date of AR5.

Six more of the 14 long series were obtained by Neukom and Gergis from crony contact (rather than public archives) including: Law Dome (Curran); ocean sediment series 106KL (Rein); four South American tree ring series: Central cluster 3A/CAN Composite 20; Central Cluster 3C/CAN Composite 25 and Central Cluster 3B/CAN Composite 23 (all said at Neukom’s website were said to be “unpublished” and to be only “available on request for LOTRED-SA contributors” and Lenca was said to be “Not public”.) The Aculeo temperature reconstruction from pigments is only available in a smoothed version of the regression product – the underlying data has not been archived. In the case of the South American tree ring series, there are public archives of measurement data that appear to overlap the versions used here. The existence of different versions makes the need for careful archiving of versions as used all the more important.

Despite IPCC’s puff about “progress”, a more objective assessment of actual progress in the field would be to inform readers from other fields that there has actually been negligible progress since AR4 in the development of new long proxies.

Tree rings: the Lenca series ends in 1987; the three long Central Cluster/CAN Composite series are Fitzroya cupressoides (FICU) series that end between 1990 and 1995. One of these series is almost certainly a version of the Rio Alerce FICU series used in the AR3 reconstructions Jones et al 1998 and Mann et al 1998-99. At most, the Neukom and Gergis versions appear to be re-processing of data collected in the early 1990s.

Ice cores: the “14″ series of Neukom and Gergis appear to include five ice core series, none of which are new since AR4 (and again, most are older than that.) Two series from the 1993 Siple Dome core appear to be included (Na- Mayewski et al 2004; and dD – Steig and White, unpublished and unarchived); a Law Dome series (presumably related to the series illustrated in Jones and Mann 2004); a series from Berkner Island (Mulvaney et al 2002); and Quelccaya, Peru (Thompson 1983, updated 2003). The Law Dome series was definitely available for AR4. As an AR4 reviewer, I had asked that it be included in the illustration of actual SH proxies. IPCC correspondence about the decision not to show this proxy (which has an elevated MWP) is in the Climategate dossier. CRU’s Tim Osborn (also Lead Author in AR5), knowing that the proxy showed an elevated MWP (and that I knew this and that I knew that they knew that I knew…), proposed that they cover themselves by mentioning the proxy in the running text but not show it in the diagram, a device readily agreed to by IPCC authors reluctant to show a SH series with an elevated MWP.

Coral: the discontinuous Palmyra series of Cobb et al (2003). See 2006 CA discussion here.

Sediments: ocean sediment 106KL was already available in Rein (2004), not Rein (2007) as shown by Neukom and Gergis. The Aculeo pigment reflectance series of von Gunten et al 2009 is new.

As I presently understand the matter, only one of the 14 data sets is “new” since AR4: the Aculeo pigment reflectance series. And while the development of novel proxies is a good idea, until the pigment reflectance methodology is replicated in other lakes, little weight should be placed on this result (particularly without the underlying data being available for analysis.)

In my opinion, a candid assessment of progress in the field by IPCC would clearly state that (1) Neukom and Gergis’ list shows that there has been essentially no development since AR4 of SH proxies that permit comparison of the medieval and modern periods; and (2) that much (far too much) of the limited data still remains unarchived. Unfortunately, WG1 Co-Chair Stocker seems more concerned with ensuring confidentiality of IPCC drafts and internal correspondence.

Stocker’s Earmarks

In December, the WG1 TSU of the IPCC sent me a formal notice asking me to remove Climate Audit discussion of the IPCC Zero Draft. In this notice, they stated:

It has come to our attention that several Chapters of the Zero Order Draft (ZOD) of WGI AR5 are being cited, quoted and discussed on the blog that you host, Climate Audit, despite the fact that each of these chapters is clearly marked “Do not cite, quote or distribute”. We would respectfully request that you remove the relevant parts with discussions of the ZOD from your blog and, furthermore, that this does not happen with the FOD.

I’ve been mulling over how to respond. I was not a reviewer of the Zero Draft and had not made any personal agreements with IPCC as a condition of receipt. I had registered as a First Draft reviewer but have not downloaded any documents in this capacity as yet.

In preparing a response, I’d been wondering what authority, if any, was possessed by WG1 or its TSU that entitled it to require or request removal of this discussion from Climate Audit. I’d looked at IPCC Policies and Procedures in connection with previous CRU requests. The procedures used in AR4 (see here) had said that the “review process should be objective, open and transparent” and did not contain any language that specifically granted authority to the TSU of a Working Group to prohibit discussion in public of its draft reports. If anything, the overriding objectives of openness and transparency would seem to support such discussion – a process that seems entirely healthy to me and one that would actually enhance the IPCC.

It turns out that Phil Jones and Thomas Stocker, Co-Chair of AR5 WG1, both agreed with my interpretation of IPCC rules on this point i.e. that the Working Groups lacked specific IPCC authority to insist on confidentiality of their drafts, and that they had, behind the scenes, taken steps to change IPCC rules to authorize Working Groups to do so. Jones’ initial contacts with Stocker on this matter are documented in Climategate 2 and arose from Jones’ reading of Climate Audit posts advocating openness and transparency by IPCC – efforts that both Jones and Stocker opposed.

I only became aware of their actions recently as a result of an IPCC cease-and-desist letter to Galloping Camel, which had posted an excellent collection of WG1 and WG2 sources. To my considerable surprise, the IPCC letter to Galloping Camel contained a quotation from IPCC Policies and Procedures here (bolded below) that contained an endorsement of confidentiality that was absent in the AR4 polices. They wrote:

The IPCC Procedures in Article 4.2 of the Principles Governing IPCC Work state that “The IPCC considers its draft reports, prior to acceptance, to be pre-decisional, provided in confidence to reviewers, and not for public distribution, quotation or citation.” (http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a-final.pdf). We therefore request the immediate removal of the ZOD chapters from your website.

This language was definitely not in the AR4 version. Indeed the document linked in the Galloping Camel letter was time stamped January 10, 2012(!). When had the new language been introduced? And by what authority? Tracing the language led to a remarkable story.

The language was almost singlehandedly introduced by Stocker (after being involved by Phil Jones.) Complicating Stocker’s efforts to obtain official sanction for enhanced confidentiality was the lack of interest in this topic by the Interacademy Panel, which had been commissioned by IPCC to review its policies and procedures. Not only did its report not contain the recommendations sought by Stocker and Phil Jones, it re-iterated the importance of openness and transparency. Nor had the language sought by Stocker been recommended in any of the numerous documents on IPCC procedures up to the second week of April 2011, less than four weeks before Stocker’s language was adopted at the Abu Dhabi IPCC meeting in May 2011.

Despite these obstacles, Stocker emerged from the IPCC plenary with his objective. It’s a long story.

Nature and the Inundation Legend

Climategate 2.0 emails shed remarkable light on the role of Nature news “reporter”, Olive Heffernan, in the development of a “legend” to place CRU data obstruction in a better light.

They show that Jones had candidly admitted to Heffernan that his real reason for refusing data was simply to obstruct potential critics – a position essentially unchanged from his notorious 2005 refusal on the grounds that “we have 25 years invested in this”. Instead of reporting Jones’ admission, Nature attributed Jones’ data obstruction to an imaginary “inundation” of data requests following an initial supply of data in 2002.

In today’s post, I’ll review the development of the legend and Heffernan’s role in the process.

Toronto 1912

Ross McKitrick, in his non-climate life, writes from time to time on particulate matter pollution in Ontario. The Toronto Globe and Mail ran a a story a few days ago about Toronto in 1912, showing the picture at left in its print edition. The amount of pollution looks like some present-day images of Chinese cities. It is also a challenge to those who think that pollution in Toronto (or similar cities) is at “unprecedented” levels.

In the late 19th century and early 20th century, Toronto’s primary form of home heating was coal. There was a substantial changeover to oil heating in the mid-20th century. The trans-Canada natural gas line reached Toronto in the 1950s and Toronto’s home heating is now mostly natural gas. Ontario’s history on this issue is probably similar to many North American (and perhaps European) jurisdictions.

Dr Phil, Confidential Agent: Re-visited

In today’s post, I continue my re-appraisal of various untrue statements made by the University of East Anglia in order to avoid disclosure of CRUTEM station data. I do not consider motives at this time. Also see preceding posts here, here.

In East Anglia’s response to July 2009 FOI requests for alleged confidentiality agreements (here) , CRU stated that, since the 1980s, they had entered into confidentiality agrements that prohibited them from providing station data to third parties, but were unable to “locate” any such agreements noting that they had “moved offices several times during the 1980s”:

Since the early 1980s, some NMSs, other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and the original station data are not passed onto third parties. Below we list the agreements that we still hold. We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s. Some date back at least 20 years. Additional agreements are unwritten and relate to partnerships we’ve made with scientists around the world and visitors to the CRU over this period. In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties. We included such statements as standard from the 1980s, as that is what many NMSs requested.

In East Anglia’s submission opposing appeals by Jonathan Jones and Don Keiller (e.g. here), they made the even stronger claim that national meteorological services(NMSs) “invariably” released information only under licences that prohibited transfer to third parties:

It was invariably the case that most NMSs only released information under licences, both written and verbal, that prohibited the further transfer of the information. There was no standard form for such licences but they were all similar in that they prohibit the onward transmission of the data to third parties.

CRU’s longstanding relationship with the US Department of Energy and its transmission of station data to the US Department of Energy (which placed CRU’s station data online (e.g. here) is inconsistent with their claims that they entered into binding confidentiality agreements in the 1980s. In my 2009 appeal, I confronted East Anglia with this relationship (a matter that they discussed in email 2929).

Their refusal of my appeal was dated Nov 12, 2009 (one day before the last email in the Climategate dossier) and was sent to me on Nov 18, 2009. While the dates of these refusals have attracted some commentary, relatively little attention has been paid to the content of the refusal (reported at CA on Nov 21, 2009 here) in which East Anglia stated that the “restrictions” applying to the station data only arose after CRU provided data to the US Department of Energy (along the lines of a scenario that I contemplated in a CA post of Aug 4, 2009 entitled Dr Phil, Confidential Agent):

In regards the information provided to the US Department of Energy, my investigation has revealed that this was done in the early 1990s prior to the imposition of the restrictions now pertaining to the data pursuant to a contractual obligation at the time. Therefore, the analogy you are drawing does not apply to the data that is the subject of this request.

Despite this explicit admission, CRU’s claims that they have been subject to longstanding confidentiality agreements have been accepted by Nature and the climate science community, which has, in turn, denounced CRU critics for not believing in the existence of actionable CRU confidentiality agreements from the 1980s.

In preparing today’s post, I considered not merely the (limited but relevant) new information from Climategate 2.0, but also re-examined contemporary (1980s, 1990s) statements about provenance of station data. Prior to 2004, there is little to no evidence of the commitment to secrecy said by East Anglia to have been “invariably the case”. Quite the contrary. Prior to 2004, there is considerable evidence of a longstanding practice of making station data publicly available (both by GHCN and CRU).

Evading Mosher’s FOI

Climategate 2.0 emails contain an interesting backstory on East Anglia’s evasion of Steve Mosher’s request for something as simple as university policies that governed entry into confidentiality agreements. Palmer consulted university specialists, receiving an answer that was adverse to the line that they were taking in their CRUtem refusals. Rather than providing this information to Mosher, Palmer adopted a tactic borrowed straight from Sir Humphrey. He pretended that he didn’t understand the question and asked Mosher for clarification – undoubtedly on the off-chance that Mosher would not return the ball. Palmer’s tactic succeeded. They avoided answering the question. The Climategate 2.0 backstory, especially Jones’ candid answers, make fascinating reading, as it shows that there were indeed compulsory university policies which were related to a term of standard employment contracts – information provided directly to Palmer.
.