Documentation - Data I/O & Storage

On the Initial GSICS Data Flow Diagram

This figure shows an initial concept for the relationship between data and the GSICS intercomparison processes. The boxes indicate file types and the ellipses indicate the four generic processes that GSICS expects to undertake.

The first process, "Find_Collocation", identifies pixels from the source data that are roughly collocated. Since GSICS primarily does inter- comparisons and intercalibrations between two data sources at a time, the data flow diagram provides a generic description of that fact in the two file types identified as "Calibrated_Geolocated_Data_Source". The Data Management Working Group also noted that the "Find_Collocation" process also needed "Orbital_Predictions" and "Collocation_Criteria" in order to do its job. Because this Data Flow Diagram is generic, it should be possible to adapt any intercomparisons that involve two instruments to this description.

The output from the "Find_Collocation" process is identified as a file type called "Collocated_Data". In the DMWG discussions, "Collocated_Data" might refer to a file type that contained a subset of the original data or might just contain indices or pointers back to that data. Which of these alternatives describes the actual "Collocated_Data" files best is left to the Research Working Group to decide.

The second process is "Transform_Data". In this step, software makes data from the two sources comparable. Thus, this process may include gap filling, spectral convolution, spatial convolution, and removal of non-random sources of error. Of course to perform these functions, "Transform_Data" needs to include "Instrument_Characteristics", such as spectral response functions, point spread functions, and noise characterizations.

The output from the "Transform_Data" step is a file type identified as "Comparison_Data". As with "Collocated_Data", this file type could actually include comparable data, or pointers to the comparable data. Of course, the GSICS intercomparisons will be easier if these data are organized so that they are accessible first by the geolocation opportunity points and then by the transformed data from each instrument. Thus, we anticipate that these data files will probabily be lists of opportunity points, with each point having a list of transformed data from instrument 1 and another list of transformed data from instrument 2.

The third process is "Filter", meaning that only some of the "Collocated_Data" are selected, so that the output from the process are refined Comparison_Data with respect to specific criteria. Again, the DMWG identified that the filtering process may need information about the scene, as well as "Filtering_Criteria". "Scene_Type" information might include a land-ocean mask, a vegetation type mask, a cloud mask, and, perhaps, even an aerosol property mask.

The output from "Filter" is the "Analysis_Data" which is the data the producer believes is most suitable for the statistical analysis that follows. The Research Working Group can specify the contents of the "Analysis_Data" files, although it seems reasonable to expect that this highly refined file type would contain geolocation and time for each intercomparison, the selected "Collocated_Data" elements, as well as some indication of which criteria helped in the selection.

The fourth and final process is "Statistical_Analysis", which provides the "Statistical_Results" of an intercomparison. These results might include scatter plots, regression statistics, trends, or almost any output from statistical libraries, such as those used by standard COTS software.

The GSICS DMWG will also provide a data dictionary with suggestions on metadata fields and conventions for such items as parameter names. When the file contents are at least generically identified, it will be much easier for the GSICS team to exchange data and algorithms and to ensure traceability of the intercomparison results.