Learn the basics on statistics you'll need to understand how your Clarity devices' are performing
This article will walk you through two common statistical metrics used to evaluate your device’s performance, R² and RMSE. Interpreting these two statistics together will help you understand your devices' performance. Read more about how to do that here.
Pearson correlation coefficient (R²)
The most common metric you’ll find when trying to understand a sensor’s performance, is the Pearson correlation coefficient, which is more commonly referred to as R², or “r-squared.” As its name suggests, this is a measure of how well correlated two datasets are. In our case, R² is used to quantify how well measurements from a sensor correlate with measurements from a reference monitor when both devices are monitoring the same location over the same period of time.
You can think about R² as telling you how well a sensor tracks changes in pollutant concentrations as compared to a reference monitor. In other words, when a reference monitor detects that concentrations are going up, can the sensor tell that concentrations are going up as well?
R² values range from 0 to 1 and are unitless. An R² close to 1 indicates a good correlation, as shown in the images below. Note that when you plot data from the two monitors on a scatter plot, there is a clear pattern, and the points lie roughly along a line.
Example of a good R². Measurements time series and scatterplot for an imaginary sensor device under analysis and reference instrument that are measuring PM2.5 over time at the same location.
An R² close to 0, like in the image below, means poor correlation and that the low-cost sensor’s data doesn’t track well with data from the reference monitor. In a scatter plot it’s hard to pick out any clear trend, as the points seem distributed randomly.
Example of a poor R². Measurements time series and scatterplot for an imaginary sensor device under analysis and reference instrument that are measuring PM2.5 over time at the same location.
Knowing your sensor R² can help you understand how to use the monitoring data you collected: if your sensor has a high R², you can trust that if it measures increasing or decreasing values, this indeed reflects an increase or decrease of pollutant concentrations on the field. You can also trust that the increase or decrease shown by your sensor measurements is proportionally consistent with the actual increase or decrease of pollutant concentration in the air. For example, if your sensor today measures double the pollutant concentration of yesterday, with a sensor with good R² you can be confident that in fact the pollutant concentration doubled.
Root Mean Square Error (RMSE)
The next statistic we’ll cover is Root Mean Square Error, or RMSE. While R2 tells you about correlation between two datasets, RMSE tells you about the difference between them. In our case, it is used to quantify the error between measurements from a sensor and measurements from a reference monitor when both devices are monitoring the same location over the same period of time. More specifically, RMSE indicates the average magnitude of the difference between the values measured by the two devices, serving as a measure of how closely the sensor measurements align with the actual values. RMSE values range from zero to infinity and carry the same unit of the pollutant concentration measured. A low RMSE means less difference between sensor measurements and reference monitor measurements and better performance.
A high RMSE means less difference between sensor measurements and reference monitor measurements and worse performance.
One benefit of using RMSE is that it carries the same units as the pollutant concentration you are measuring, which makes it easy to interpret. For example, if you are comparing PM2.5 measurements from a low-cost sensor and a reference monitor, a RMSEof 5 µg/m3 means that, on average, the low-cost sensor measurement was 5 micrograms off from the reference monitor.
For example, if your RMSE was 5 ug/m3 and your device is measuring 59 µg/m3, then it is likely that the actual concentration is most likely between 54 - 64 µg/m3.
Knowing your sensor RMSE can help you understand how to use the monitoring data you’ve collected. For the sensor in the example above, if you see an increase of 3 µg/m3 in your sensor measurements you should question whether this reflects true increase in pollutant concentration, since you know that your sensor can be off by 5 µg/m3 from the true value: you might be looking at “noise”. However, if you see an increase of 15 µg/m3, you can have more confidence that you detected an actual increase in pollutant concentration, since the increase is greater than your sensor RMSE.
Root Mean Square Error (RMSE) vs Mean Absolute Error (MAE)
Mean Absolute Error (MAE) is another metric often used to quantify by how much your sensor measurements deviate from reference monitor measurements. The difference between RMSE and MAE lies in how the two metrics are computed: RMSE (Root Mean Square Error) represents the average square root of the squared differences between the measured values and the true values, while MAE (Mean Absolute Error) represents the average of the absolute differences between the measured values and the true values. This can seem confusing, but do not worry: the difference is subtle, and in the context of evaluating the performance of sensors the two metrics can be used interchangeably, as they very often evaluate to about the same value.