Expert Developer Guidance

The following was contributed by Chris Atkinson, UK Met Office, September 2020:

HadIOD (Met Office Hadley Centre Integrated Ocean Database) is a database of global historical in situ ocean temperature and salinity observations.  It brings together observations made by surface-only observing platforms (like ships and buoys) and observations from sub-surface ocean profiling platforms (like profiling floats and bathythermographs) and supplements these with metadata including quality flags, bias corrections and estimates of measurement uncertainty.  ‘Integrated’ refers to the merging of data from sources concerned with different components of the observing system (e.g. the surface or subsurface ocean) into a single data store, and the inclusion of appropriate metadata (such as bias corrections and duplicate flags) to enable greater internal consistency.

Data coverage is global from 1850-present.  Monthly updates are made on a best endeavours basis with approximately a few months delay.  Data are extracted from the database into CF-compliant NetCDF format for release.  The data and a detailed user guide can be obtained from https://www.metoffice.gov.uk/hadobs/hadiod/.  Releases are versioned and the current version is HadIOD.1.2.0.0.  Published information (describing HadIOD.1.0.0.0) can be found in Atkinson et al. [2014].

The product is targeted at activities wanting to ‘pick up and use’ (ungridded) climate-quality ocean observations e.g. ocean reanalyses.  The focus is on providing the essential information needed for these activities in a convenient format.  It is important that users of ocean data correct for known biases (which can introduce artificial variability in the data) and explore the sensitivity of their applications to observational uncertainties.  By combining this information in a single data set, we allow users to more easily utilize current understanding of observation error in their applications.  Less focus is given to providing the comprehensive observation metadata found in some observation collections (e.g. data provenance) and as such the product may be of lesser interest to observation specialists.  Note that further metadata and IDs linking the observations to their source data collections are available in our own internal database but not currently in the files we release, users can contact us if these are of interest.    

Observation errors are modelled as the true value plus a bias associated with the type of platform/instrument, a bias specific to an individual platform/instrument and a random measurement error.  Where possible bias corrections are provided to reduce these biases.  Uncertainty estimates are provided for the residual biases in the corrections and for the random error.  The extent to which this information can be provided for each observation varies by observation type dependent on current knowledge.  With applications such as data assimilation in mind, we try to provide at a minimum some estimate of uncertainty and an overall quality flag for every observation in HadIOD.  Platform/instrument type bias corrections are currently provided for temperature observations made by ships and XBT/MBTs.  The uncertainty in these bias corrections can have complex structures in space and time.  To facilitate exploring this uncertainty, multiple sets of bias corrections are provided that can be used interchangeably.

Although HadIOD estimates observation errors, users will need to consider whether other sources of error are present in their application.  For example, sampling errors may arise if an observation does not exactly represent the quantity of interest (e.g. if a point observation is used to represent an area average).  Furthermore, although HadIOD provides estimates of observational uncertainty, these do not capture the structural uncertainty arising from the fundamental choices made in the dataset construction process (e.g. observation error model used, bias correction schemes used, handling of missing metadata, quality control applied, data collections included).  Consequently, the uncertainty in any analysis based on HadIOD may be underestimated and non-climatic features may remain in the data.  Users should consider also using alternative datasets in their analysis to gain a broader understanding of the uncertainties, for example a user might consider blending HadIOD surface data with a different source of profile data or an alternative source of historical SST bias corrections.  Further exploring structural uncertainty is something we may consider for future versions of HadIOD.

It is emphasised that HadIOD brings together observations and metadata from a variety of sources and as such relies on the work of many other people.  The sources of observations in the HadIOD.1.2.0.0 public release are: ICOADS.2.5.1 [Woodruff et al., 2011], EN4 [Good et al., 2013] and CMEMS [http://marine.copernicus.eu/] drifting buoy data.  EN4 in turn draws on sources including the World Ocean Database [Boyer et al., 2013; http://www.nodc.noaa.gov/OC5/WOD13/], Global Temperature and Salinity Profile Project [http://www.nodc.noaa.gov/GTSPP/] and the Argo project [http://www.argo.net]. 

Several alternative observation datasets to HadIOD are available, including those used by HadIOD.  The objectives and features of each dataset vary.  HadIOD is intended to complement existing data collections and its key strengths and limitations are given below.  A possible alternative to the HadIOD dataset is CORA [http://www.coriolis.eu.org/Data-Products/Products/CORA] which also provides surface and subsurface temperature and salinity observations in a single dataset and is aimed at similar users.

Key Strengths:

• Blending of temperature and salinity observations from the surface and subsurface ocean in a single product. 
• Long record (1850 to present) with regular updates.
• Observation bias corrections, uncertainty estimates and quality flags.
• Data release targeted at users wanting to just ‘pick up and use’ climate-quality ocean observations appropriately.
• Detailed user guide (including guidance on working with bias corrections and measurement uncertainties).

Key Limitations:

• Data coverage can be sparse; in general, the oceans are better sampled in space (horizontally) and depth as time progresses (summary plots are available in the user guide).
• Improved or new versions of some underpinning data/metadata sources are now available (e.g. ICOADS 3.0, latest ship/XBT/MBT bias corrections).  Surface salinity content could be improved by adding other sources.  These will be included in future releases.
• Less metadata provided than for some observation collections (by design), observation specialists may find other datasets more suitable for observation study (further metadata and IDs linking the observations to their source collections are available in our database but not in the files we currently release, users can contact us if this is of interest).
• Structural uncertainties (the fundamental choices made during dataset construction) are not fully explored, users should consider using other datasets as well.