Course outline

Lesson 5: Understanding your Calibration Results

By the end of this lesson, you’ll understand:

  • The typical performance Clarity expects and uses to evaluate calibration
  • How to understand the long-term performance of your devices after calibration
  • Seasonal changes in NO2 performance

Developing performance targets for low-cost air quality sensors is an ongoing effort across governments and academic institutions, and poses several important challenges. You can read more about the history and current state of performance target for low-cost sensors in the article here.

The required sensor performance varies depending on the specific use cases of the collected data, which can significantly differ from one project to another. Additionally, low-cost sensors inherently have limitations that need to be addressed. While processes like calibration can help partially overcome these limitations, there exists a tradeoff between fine-tuning the sensor's performance and managing the operational costs and complexity of air quality monitoring projects. Moreover, sensors do not perform equally well in different environments and struggle to deliver accurate results under extreme environmental conditions, and performance targets need to take this into account as well. Finally, the evaluation of sensor performance metrics is dependent on the conditions under which the tests are conducted. This dependence creates challenges in ensuring consistent and comparable evaluations across various regions and projects. 

Due to these complexities, Clarity has taken a proactive approach by developing typical performance metrics that we use to evaluate calibration results. These typical performance metrics have been established based on our experience in the field and are designed to ensure that the data collected from the devices in real-world applications are both usable and reliable. By striving to meet these performance targets through calibration, Clarity aims to enhance the quality and accuracy of air quality measurements, thereby enabling more effective decision-making processes based on the collected data.

Typical performance metrics are shown in the table below:

  R2 RMSE
PM2.5 1-hour mean mass concentration ≥ (greater than or equal to) 0.6 ≤ (less than or equal to) 8 µg/m3
NO2 1-hour mean concentration ≥ (greater than or equal to 0.5 ≤ (less than or equal to) 10 ppb

 

These statistics will be used as a benchmark by your Environmental Project Manager to determine whether the performance of your calibrated devices meets our standards. In the calibration report sent to you, you’ll be able to see the R2 and RMSE of your devices and how they measure up relative to these targets.

The calibration report will tell you about how your devices performed during the collocation period itself. Maintaining a long-term collocated device will help you understand performance of your devices throughout the remainder of your project.

To learn more about performance of sensors without collocation, you can read the article here.

If you want to learn more about R2, RMSE and how to interpret them together, check out the article here.

Understanding detection limits

Detection limits are important to understand as you begin interpreting your data from any air monitor. They are the levels below which the instrument stops being able to measure a given pollutant accurately. If data reported from your device falls below the detection limit, it may be appropriate to understand that measurement as ‘low’, but it would not be scientifically accurate to use the measurement itself in a given analysis.

The OPCs used inside of Clarity Node-S devices to measure PM2.5, have a detection limit of ~5 µg/m3, so it is not suitable for measuring very low ambient concentrations (0-10 µg/m3). The ECSs used to measure NO2 have a detection limit of ~10 ppb, so they cannot measure accurately in low ambient ranges of ~0-15 ppb.

So - if you know ahead of time that concentrations in your project area regularly fall below these thresholds, it may be difficult to get useful data out of the Clarity Node-S. If you were to measure concentrations below 10 µg/m3 PM2.5 or 15 ppb NO2, you would be able to confidently say that those measurements were, in fact, low and likely truly below those levels. However, you would not be able to confidently report the true concentration. You could not say, for example, that one device measured 8 µg/m3 while another measured 5 µg/m3.


Understanding long-term performance of your devices

The calibration report tells you how your devices performed during collocation, but how about long-term performance once the devices are in the field?

The calibration performance metrics are expected to hold true as long as the sensors are operated within the same environmental and pollutant composition conditions that were experienced during calibration.

PM performance often remains stable over time, unless there is a change in the particle composition (e.g. seasonal crop burning, periodic wind-blown dust, etc). That means that in many cases the performance seen during your collocation period and presented in your calibration report will be representative of the long-term performance of your devices.

NO2 performance, in contrast, may vary throughout the year, especially if the environmental conditions at the project site diverge from the calibration conditions. When your devices are experiencing similar temperature and relative humidity conditions as when you collocated them, you can expect that they will perform similarly to the performance seen during the collocation period and presented in your calibration report. As you move to different seasons, with different temperature and relative humidity ranges, the performance will shift. In general, NO2 sensors perform poorly in hot and dry conditions, so in many regions, a dip in performance is expected in the summer months. If you have a permanently collocated device, you can use that device to understand how the performance of the whole network varies throughout the year. Learn more about how to check the performance of a collocated device here.

An example of seasonal changes in NO2 performance, with lower R2 values in the Summer months.

Other Learning Resources: