Expert Developer Guidance

Least squares procedures of optimal estimation, when applied to gappy and erratic data, result in the solutions which predominantly project onto the most energetic patterns of a priori error covariance. This property of the solution allows to combine the classical least squares technique with the approach of a space reduction in order to develop a computationally effective procedure of objective analysis for observed historical climate data. (Such data are characterized by comparatively precise observations and good coverage in the last few decades, and poor observational coverage prior.) An important aspect of our approach is that it also produces verifiable error bars for analyzed values. For details of the technique see Kaplan et al. 1997, for a simple qualitative introduction see Kaplan et al. 2001.

Here we applied the reduced space optimal smoother technique to the U.K. Meteorological Office observational data set of historical sea surface temperatures (SST) (Bottomley et al 1990; Parker et al 1994). The primary difference between the British data set and the COADS SST is that the former was corrected for the systematic biases in bucket measurements of SST before 1940s (Folland and Parker, 1995). The details of our analysis (http://ingrid.ldgo.columbia.edu/SOURCES/.KAPLAN/.RSA_MOHSST5.cuf/.OS/.ssta/) and its verification are given in Kaplan et al. 1998, but there are a few points I'd like to make here:

1. Weak feature of this product is that it uses ship data only. It doesn't use buoys or satellites. So if you can get away with using only data after 1981, you should use Reynolds' OI SST instead -- it is much better (and of 1 degree spatial resolution!), since it uses buoys and AVHRR satellites data.

1. Another applicational disadvantage of this product is that it is of sparse resolution and globally incomplete (and has no sea ice information). So if you are a modeler who seeks an SST analysis to force an atmospheric GCM, you'd rather get HadISST1 analysis by the U.K. Met Office (Nick Rayner et al.), which is of 1 degree resolution, globally complete, accompanied by sea ice concentrations -- a modeler's dream which was particularly produced with the ECMWF Reanalysis project in mind.

1. So what is our product good for and who would be using it? Strong part of this product is that it is produced in a conceptually integral and simple objective way under a set of conservative assumptions from the (allegedly quite poor) data. Potential users are people who need long records of the SST for statistical analyses, and care mostly for the large-scale climatic connections in the data. I'd like to stress here that 4-5 degree spatial resolution is pretty much all one can reasonably derive from the SST data prior to 1950s. And except for a few places with particularly high historical sampling the existing 1 degree resolution analyses (like HadISST1) are bound to interpolate linearly from a sparse resolution analysis anyway. So our product is for users who want some degree of spatial uniformity throughout the analysis period (and also for those who need error bars).

1. By popular request we even started providing monthly extensions for our ship-based analysis by the Reynolds' OI via projecting the OI fields to our set of basis functions which is used for the entire analysis period (1856-present):http://ingrid.ldgo.columbia.edu/SOURCES/.KAPLAN/.EXTENDED/.ssta/ That way the long records of SST up to the present are provided without jumps in the effective resolution when the satellite data kicks in.

1. A weakness of this product which it shares with all least squares based analyses of gappy and erratic data is that the analysis itself is spectrally redder and of smaller amplitude than the true signal, and this discrepancy in properties becomes stronger as the observed data becomes poorer (Kaplan et al. 2001). Since the data gets worse as we go back in time, the analysis might display some apparent shifts in spectral and energetic properties which in fact reflect data availability, not the true physical SST changes.

1. There is also an important issue of SST trends. Jim Hurrell and Kevin Trenberth in their 1999 paper suggested that the analysis which conservatively assumes the stationarity of the mean SST climatology (like ours) might be underestimating the long-term changes in the SST, and argued that it is better to figure out the long-term variability beforehand and prescribe it a priori (e.g. like it is done in the HadISST1). Their criticism regarding long-term trend underestimation is certainly valid, as our preliminary study has shown; however, prescribing the global trend a priori might result in overestimation of it by the analysis -- not a good outcome either (Kaplan et al. 2001). We are currently working on the procedure which would allow us to separate long-term and interannual variability, obtain objective estimates of both and recombine them for a better analysis which even Kevin would approve!

1. After you are done with your study using this product, it would be useful to look at the large-scale time-dependent analysis error in your domain of interest (http://ingrid.ldgo.columbia.edu/SOURCES/.KAPLAN/.RSA_MOHSST5.cuf/.OS/.err/), also to look at the number of observations which went into the analysis (http://ingrid.ldeo.columbia.edu/SOURCES/.KAPLAN/.RSA_MOHSST5.cuf/.Nobs/), and to try to imagine how the uncertainty could affects your conclusions. Be particularly alerted if the key periods of changes in your conclusions coincide with periods of particulaly bad data, fast changes in the coverage, etc.

Alexy Kaplan, Columbia University, May, 2001###