Climate Data Analysis Tools and Methods

Overview: Climate Data Processing
The following is by Dennis Shea (NCAR): The first rule of data processing is to look at your data! What data format is being used? What variables are available and what are the units? What type of grid: rectilinear, curvlinear or unstructured? What is the orientation of the grid: north-to-south or south-to-north? What longitude order? What temporal and vertical coordinate(s) are used? Are... more
Statistical & Diagnostic Methods Overview
The following is by Dennis Shea (NCAR): By definition, climate is the statistics of weather over an arbitrarily defined time span. The methods used to derive the statistical estimates can be simple or very complex. The most common statistic is the average of some variable (eg., temperature). However, soley focusing on the average can be misleading. For example, the average temperature may be... more
The following is by Dennis Shea (NCAR): Binary files encountered in climate are generally created from compiled languages such as fortran or C/C++. These are easy to create and are supported by the compiled languages. Some software tools used in climate research (IDL, Matlab) can produce binary files but they are proprietary and are best read with the tools that produce them. ... more
Climate Data Processing Software
The following is by Dennis Shea (NCAR): A common question: "What is the best software to use for climate data processing?"There is no simple answer. All software tools and languages have strengths and weaknesses. For large scale data processing on a variety of data sets in assorted data formats and differing project requirements, it is unlikley that a perfect tool or language exits. Often, a... more
Common Climate Data Formats: Overview
The following is by Dennis Shea (NCAR): Data formats commonly encountered in climate research fall into 3 generic categories: GRIB, netCDF and HDF. All of these formats are portable (machine independent) and self-describing.  Self-describing files can be examined and read by the appropriate software without the user knowing the file's structural details.  Further, additional information about the... more
Empirical Orthogonal Function (EOF) Analysis and Rotated EOF Analysis
The following is by Dennis Shea (NCAR): Empirical Orthogonal Function (EOF) analysis: In climate studies, EOF analysis is often used to study possible spatial modes (ie, patterns) of variability and how they change with time (e.g., the North Atlantic Oscilliation). In statistics, EOF analysis is known as Principal Component Analysis (PCA). As such, EOF analysis is... more
The following is by Dennis Shea (NCAR): GRIB is the format used by all the world's operational weather centers. A brief (edited) description of GRIB: "The World Meteorological Organization (WMO) Commission for Basic Systems in 1985 approved a general purpose, bit-oriented data exchange format: GRIB (GRIdded Binary). It is an efficient vehicle for transmitting large volumes of gridded... more
The following is by Dennis Shea (NCAR): The Hierarchial Data Format is available in two  versions: the original HDF4 and the more recent HDF5. Unfortunately, HDF4 and HDF5 interfaces and data models are completely incompatible.  The HDF5 data model is more flexible and is a "a true hierarchical file structure, similar to the Unix file system." HDF5 does have some new features that are appealing... more
netCDF Overview
The following is by Dennis Shea (NCAR): NetCDF (Network Common Data Form) is designed to facilitate access to array-oriented scientific data. NetCDF is a portable, "self-describing" format. This means that there is a header which describes the layout of the rest of the file, in particular the data arrays, as well as arbitrary file metadata in the form of name/value attributes. The additional... more
Regridding Overview
The following is by Dennis Shea (NCAR): Regridding is the process of interpolating from one grid resolution to a different grid resolution. This could involve temporal, vertical or spatial ('horizontal') interpolations.  However, most commonly, regridding refers to spatial interpolation. There are numerous grid interpolation methods and users should choose the method appropriate... more
Taylor Diagrams
The followng is by Dennis Shea (NCAR): Taylor diagrams provide "a concise statistical summary of how well patterns match each other in terms of their correlation, their root-mean-square difference and the ratio of their variances."  Additional information such as percent bias can be added to the conventional Taylor diagram (see below).  The Taylor diagram provides a graphical framework that... more
Text (ASCII) files
The following is by Dennis Shea (NCAR): Text files encountered in climate research are typically structured as a sequence of lines terminated with a special 'end-of-line' character (typically, the "null" character). Most climate oriented text files are in ASCII format. An advantage of text files is that they are human readable. Unfortunately, text files are very inefficient for archival... more
Trend Analysis
The following is by Dennis Shea (NCAR): The detection, estimation and prediction of trends and associated statistical and physical significance are important aspects  of climate research. Given a time series of (say) temperatures, the trend is the rate at which temperature changes over a time period. The trend may be linear or non-linear. However, generally, it is synonymous with the linear slope... more
Web Based Visualization and Processing for Climate and Weather
There are number of web-based sites that provide the cabability to visualize, perform limited proceessing tasks and, somtimes download data. Some are designed for a specific data product while others are cabable of creating graphics and performing computations  on multible data sets. Common tasks include difference maps, correlations, composites, climate monitoring and deriving climate indices.... more