HDF

The Hierarchial Data Format is available in two  versions: the original HDF4 and the more recent HDF5. Unfortunately, HDF4 and HDF5 interfaces and data models are completely incompatible.  The HDF5 data model is more flexible and is a "a true hierarchical file structure, similar to the Unix file system." HDF5 does have some new features that are appealing to climate research, such as, parallel I/O and variable compression.  (Note: Our experience is that any compression level greater that '1' is not worth the additional time required for the increased compression.)

The HDF{4/5}-EOS format extension defines three additional data types based on HDF objects: grid, point, and swath. These data types allow the file contents to be referenced to Earth coordinates, such as latitude and longitude, and to time. An irritating issue is that standard HDF library calls cannot readily access geolocation or time data or metadata. Users must use the HDF{4/5}-EOS interfaces.The converse is also true. Certain information can only be accessed via standard HDF4 interfaces. Hence, both interfaces may be required to get at all the information on the file.

The netCDF user communities have numerous conventions for creating the contents of netCDF files, in particular, the commonly used COARDS and CF conventions. Unfortunately, the HDF communities do not seem to have any generally adopted conventions for HDF files. Perhaps, the variety of data sources, particularly satellites, inhibits the creation of common conventions?

Reference:

Klein, L., A. Savtchenko, A. Taaheri, C. Praderas, and S.J.S. Khalsa, Ramachandran, B., C. Justice, M. Abrams (Ed.) (2011), The Language of EOS Data: Hierarchical Data Format. Land Remote Sensing and Global Environmental Change: NASA's Earth Observing System and the Science of ASTER and MODIS, 203-227, Springer, isbn: 978-1-4419-6748-0

Volume 11, 2011, DOI: 10.1007/978-1-4419-6749-7