Automated Data Quality Control (QC)

How Clarity identifies and flags untrustworthy measurements, enabling you to exclude them from analyses and public data sharing.

Clarity has many checks in place to ensure that data are high quality. Part of this is Automated Quality Control (QC) performed before serving or displaying data. 

QC Flags

QC Flags are used to highlight measurements that may not be trustworthy. Clarity maintains a library of Flags that have been developed based on our understanding of air pollution, sensing principles, and our sensors' failure modes. These Flags are automatically applied to metric values when specific conditions arise. Flagged metric values can then be appropriately managed to ensure that only valid data is used for real-time sharing and analysis. The library of automated QC Flags is presented in the Automatic Quality Control Flags table below.

The below table describes each field in the Automated Quality Control Flags table.

How QC Flags and Severity are Used: QC Assessment

In the Automated QC process, all metric values are presumed Valid unless flagged by Clarity’s Automated QC system. A single metric value can have multiple Flags (e.g., QC.I.OOB.001 and QC.I.OOB.002), which may have different Severity levels. To reconcile this, an overall QC Assessment is determined for each metric value by considering all Flags that are applied to it and assigning the appropriate indicator (Valid, Invalid, etc.) based on the most severe of the Flags’ Severity levels.

For metric values dependent on multiple sensor measurements for calibration, the automated QC system applies all Flags present on the inputs. The calibrated value's QC Assessment is determined by considering all Flags applied to it and assigning the appropriate indicator based on the most severe of the Flags’ Severity levels.

For aggregated metric values, such as hourly or daily averages, the Automated QC system applies all Flags present in the aggregation period. The aggregated value's QC Assessment is determined by considering all Flags applied to it and assigning the appropriate indicator based on the most severe of the Flags’ Severity levels.

Using QC Assessment

The QC Assessment can be utilized to appropriately handle data for internal and public purposes. The below table summarizes Clarity's recommended handling of the QC Assessment and also documents how Clarity handles QC Assessment on Dashboard.