iQuam - in situ SST quality monitoring

Overview
QC Algorithm
Monitor Interface
Effectiveness

iQuam was developed at NOAA Center for Satellite Application and Research (STAR). It performs three major functions, in near-real time:

Quality control (QC) of in situ SSTs
Monitoring of QCed in situ SSTs online
Serving of reformatted in situ SST data with quality level and flags appended

The real time processing is performed as follows

from Sep 1981 - Mar 2010, ICOADS data was used instead of GTS.

Ship and buoy (drifters and moorings) data come from ICOADS, NCEP or FNMOC. Real time data are refreshed every 24hrs and ingested into iQuam with a latency of 4 hours on average. FNMOC, IMOS, HR-Drifter, ICOADS and ARGO data come with their own quality flags (QFs), which are preserved in the iQuam output files. In general, QFs from data sources are not used in the iQuam QC. Also, a blacklist QF based on the Copernicus Maritime Service (CMS) blacklist compiled by Meteo France is reported in iQuam output files but not used in iQuam QC. ARGO QFs are used to select the best quality near-surface data from above 10m depth, which are further subjected to the standard iQuam QC. ARGO data includes a greylist, which is is also reported in QFs but not used in QC. A blacklist for drifters, compiled by AOML, is also used. Again, this information is stored in QFs but not used in the QC.

iQuam QC is then performed and corresponding QFs are appended (but not applied) to in situ data. All in situ data are preserved in iQuam files, and no data excluded based on iQuam or external QFs. QCed in situ data, stored in self-described NetCDF format, are available via HTTP or FTP.

Currently, all processing of the previous month's data is completed by the 15th day of the following month.

The QC algorithm implemented in iQuam consists of different steps of processing which can be categorized into five groups:

Prescreening - Processing which precedes QC checks and is aimed at resolving known data-specific historical problems (e.g., removing of duplicates and wrongly formatted data).
Plausibility check - Analyzes individual fields in the data records and relationships between them to verify their plausibility (e.g., geolocation check).
Internal consistency check - Checks different measurements from the same platform for internal consistency (e.g., tracking and SST spike checks).
Mutual consistency check - Checks nearby measurements from different platforms for consistency (also known as 'cross-platform' or 'buddy check').
External consistency check - Checks in situ SST for consistency with reference (first-guess) field (also called 'reference check' or 'background check')

Inherited QC information - QC information from external sources, e.g. ICOADS QC, Argo QC and buoy blacklists - is reported in iQuam files but not used in setting up the iQuam QC.

In addition to some basic processing steps, the 5 major steps are duplicate removal (DR), platform track check (TC), SST spike check (SC), reference check (RC) and cross-platform check (XC). Algorithms for the reference check and the cross-platform check are employed from the currently existing Bayesian QC method [Lorenc and Hammon, 1988; Ingleby and Huddleston, 2007] with minor modifications. Details can be found in the following publications:

Xu, F. and A. Ignatov, 2013: in situ SST quality monitor (iQuam), JTECH, submitted. download
Ingleby, B. and M. Huddleston, 2007: Quality control of ocean temperature and salinity profiles - historical and real-time data, J Marine Systems, 65, 158-175.
Lorenc, A.C. and O. Hammon, 1988: Objective quality control of observations using Bayesian methods. Theory, and a practical implementation. Q. J. R. Meteorol. Soc. 114, 515-543.

Flowchart of QC algorithm is as follows.

Monitoring of statistics of both QC and SST anomalies are also provided in iQuam. It includes monthly global maps, monthly QC/SST statistics, long-term time-series and individual platform statistics (track map, SST time series and performance history). In most cases, daily maps and statistics are available as well.

Interfaces are shown below.

The interface for platform monitoring:

Time series of monthly statistics (SD of Pathfinder minus in situ SST) before (left) and after (right) QC:

Version 2.1 (Mar 2019)

Merged RT GTS data from NCEP, FNMOC, ICOADS
- To improve data stability
- To address the 5-to-7 digit ID WMO transition (still not addressed in NCEP GTS)
Added Argo floats from 2 more sources (in addition to IFREMER)
- USGODAE & NOAA NODC
Replaced ICOADS R2.5 with R3.0
- Also NetCDF data are used instead of IMMA1
Added Interactive/extended plots for individual platforms
- Added time series of mean/SD/NOBS for individual platforms (All time series are now interactive)
Added hourly maps
- Useful to analyze & fill data gaps
Added permalink feature
- Useful to share the page content with partners
Files are updated once daily (vs. twice daily before)
- Increased data volumes started creating job run conflicts
Format changes of Data Files
- SD_filename variable removed
- source variable added: name of remote site data was taken from
- optional_flags1 variable
  - Bit flags have been rearranged
  - Different flags available for Argo and ICOADS
- Current optional_flags2 unchanged. New flags added for HR Drifters in previously unused bits.

Version 2.0 (Apr 2016)

Platforms: Ships, drifters, tropical and coastal moorings, Argo floats, Hi-Res Drfiters, IMOS ships, CRW buoys
QC: Duplicate Removal, Geo-Location, Travel-Speed, SST-Gradient, Reference SST, Cross-Platform, Performance-History Checks
CMS buoy blacklist, Argo heritage QFs and IMOS heritage QFs appended (only Argo QFs are used in setting iQuam2 QL)
Time interval: September 1981 - present
Data source: ICOADS , GTS , Argo in near time mode , Argo in post processing mode , Hi-Res Drifters , IMOS , CRW .
Monitoring statistics are based on best quality data only (i.e. quality_level == 5)
Data format: GDS2/NetCDF4
More information: documents under preparation

Version 1.0 (Feb 2010)

Platforms: Ships, drifters, tropical and coastal moorings
QC: Duplicate Removal, Geo-Location, Travel-Speed, SST-Gradient, Reference SST, Cross-Platform Checks
Time interval: January 1991 - present
Data source: NCEP GTS
Monitoring statistics are based on combine of best quality and good quality data(i.e. QF AND 0x0001 == 0)
Data format: HDF4.
More information: Xu, F. and A. Ignatov, 2013: in situ SST quality monitor (iQuam), JTECH

What kind of in situ SST data are included in iQuam, and where are they from?

In iQuam v2.10, the following in situ SST data types are included:

Commercial ships, drifting buoys, tropical moored buoys and coastal moored buoys are from:
- GTS (NOAA/NCEP internal ftp server)
- ICOADS v3.0
- GDP (OSMC)
- USGODAE/GDAC
Argo floats data are from
Hi-Res Drifter SST data are from E-SURFMAR.
IMOS ships are from Integrated Marine Observing System (IMOS).
Coral Reef Watch (CRW) data are from NOAA Coral Reef Watch.
Blacklist data (not used in the iQuam QC) is taken from the following sources:
- CMS Blacklist -- Copernicus Maritime Surveillance Buoy Blacklist, compiled by Meteo France
- Argo Greylist
- AOML DB (Blacklist)

What is the iQuam data file format? How do I use the data?

iQuam v2.10 data are in GDS compatible self-describing NetCDF 4 format. Refer to global and data layers attributes for more information. Suggested usage of the quality_level is:

for high-accuracy applications, use data with best quality level (quality_level == 5);
for general applications, use data with acceptable quality (quality_level >= 4);
for advanced usage of individual QC checks, refer to the definition of iquam_flags, optional_flags1 and optional_flags2.

Note that all statistics presented on this website are based on the (quality_level == 5) condition. Please be advised that the quality control is only intended for sea surface temperatures. Other measurements have not been quality controlled.

All data can be directly accessed here.

How do I read/access iQuam data files?

For browsing the data, we recommend the tool hdfview. If you prefer to use a unix command line tool over a GUI interface, we recommend ncdump, which is part of the NetCDF software library.

For accessing data files programtically, the NetCDF software library contains APIs for Fortran, C/C++, and Java. For Python, we recommend the netCDF4 module.

How do I download and visualize iQuam data?

An example Python script is available from our ftp site. This script will download an iQuam data file and produce a scatter plot of the SSTs on a world map.

Why is iQuam important for SST research?

In situ SSTs sensors are mounted onboard different platforms and maintained by different countries and agencies. Buoys remain unattended in a hostile environment for years, and ship records are subject to human errors. Additional errors occur during data transmission (to satellite and back to ground), processing and distribution via GTS. The quality of in situ SSTs is thus often suboptimal and non-uniform.

A small fraction of erroneous in situ measurements may significantly affect results of satellite Cal/Val. Non-QCed data cannot be used in Cal/Val, and removing 2-4% of bad reports significantly improves the validation statistics.

The QC practices adopted in the remote sensing community are overly simplistic, and far inferior to the more sophisticated, systematic, and well-developed procedures adopted in the meteorological and oceanographic communities. Moreover, the QC procedures are often inconsistent across various satellite SST groups, making their Cal/Val results difficult to compare.

NOAA is responsible for the maintenance and development of operational SST products from the polar and geostationary satellites. Thus a near-real time in situ SST Quality Monitor (iQuam) was developed, using community advanced and consensus QC procedures and practices, to support Cal/Val efforts for a wide range of NOAA and GHRSST SST users.

What browser and computer should I use to correctly view the figures and tables in iQuam?

Currently, mainstream browsers Firefox, Safari and Chrome are supported. A minimum screen resolution of 960*720 is recommended to correctly display this web site. If there are any remaining issues, please contact us.

How often are iQuam data updated?

iQuam v2.10 files for the most recent month are updated every 24 hrs with an average latency of 4 hours. Data may be updated for several days after it is receieved. Data from the previous month are finalized on the 15th day of the following month.

What is the meaning of the `fvxx.x` numbers in the iQuam data file names.

The fvxx.x numbers in the iQuam data file names are an indication of the number of times a new version of the data has been produced, after the nominal period of data gathering and processing has passed. During the current month, and for 4 days into the next month, data files produced for that month are given a version number of fv00.00. After that time, whenever a new data file is produced, the version number is incremented by one, e.g. to fv01.00, fv02.00, etc. Given that the file is processed at least one more time after the nominal period, final files should have a version number of at least fv01.00. Note that this code change was implemented in Apr 2017, and prior to that all files were from reprocessed data and all fv's were set to 01.00.

What are the future plans for iQuam?

Currently the following upgrades in iQuam are being considered.

Addition of new data sources, especially data from the World Ocean Database (WOD). This includes Expendable Bathythermograph (XBT) data, Mechanical Bathythermograph (MBT) data, and Conductivity-Temperature-Depth (CTD) data. Also addition of new ship, buoy, and drifter data from the Copernicus Marine Environment Monitoring Service (CMEMS), and Thermosalinographs (TSG) ship data from NOAA NODC.
Improvements to the data processing of iQuam, to increase stability and efficiency. These changes will be transparent to the user.

in situ SST Quality Monitor v2.10

Time series of monthly statistics (SD of Pathfinder minus in situ SST) before (left) and after (right) QC:

Funding

Data Sources/Providers

Partners/Users

Other Acknowledgements

Disclaimer

Papers

Presentations

Version 2.1 (Mar 2019)

Version 2.0 (Apr 2016)

Version 1.0 (Feb 2010)

What kind of in situ SST data are included in iQuam, and where are they from?

What is the iQuam data file format? How do I use the data?

How do I read/access iQuam data files?

How do I download and visualize iQuam data?

Why is iQuam important for SST research?

What browser and computer should I use to correctly view the figures and tables in iQuam?

How often are iQuam data updated?

What is the meaning of the `fvxx.x` numbers in the iQuam data file names.

What are the future plans for iQuam?

Other questions please contact us:

NOAA SST Monitoring

in situ Observations & Resources

in situ SST Quality Monitor v2.10

Time series of monthly statistics (SD of Pathfinder minus in situ SST) before (left) and after (right) QC:

Funding

Data Sources/Providers

Partners/Users

Other Acknowledgements

Disclaimer

Papers

Presentations

Version 2.1 (Mar 2019)

Version 2.0 (Apr 2016)

Version 1.0 (Feb 2010)

What kind of in situ SST data are included in iQuam, and where are they from?

What is the iQuam data file format? How do I use the data?

How do I read/access iQuam data files?

How do I download and visualize iQuam data?

Why is iQuam important for SST research?

What browser and computer should I use to correctly view the figures and tables in iQuam?

How often are iQuam data updated?

What is the meaning of the fvxx.x numbers in the iQuam data file names.

What are the future plans for iQuam?

Other questions please contact us:

NOAA SST Monitoring

in situ Observations & Resources

What is the meaning of the `fvxx.x` numbers in the iQuam data file names.