How to identify outliers in a time series with correlated variables

Question

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the readings should vary slightly given the setup of the experiment.

Some sensors experience power or other issues that causes them to record bad data. I would like to use the other data recordings and the trends associated, to be able to best detect outliers / anomalies from a given sensor. Ultimately I would like to label that data NaN. An example of the data I am looking at is provided here:

As is evident, one of the sensors is having an issue and is fixed. Is there a preferred method for detecting outliers like this? Is it acceptable to just do an IQR using all the data? I am working in Python with Pandas for my data analysis.

Answer 1

I suggest a simple "voting" mechanism: remove top and bottom n readings from each observation. This is often used in scoring of sporting events, such as diving where out of 10 judges top and bottom 2 scores are removed from calculations.

This is not the most optimal approach, of course, especially if it's rare that the sensor is faulty. However, it's very simple and robust. Even if you decide to employ more sophisticated approach, I'd still keep this one as a benchmark.

Another approach, if to constantly impute the measurement of each sensor based on the readings of other sensors. When the imputed value is different enough from an actual measurement, it indicates the problem. Of course, you'd have to take into account, that when the faulty sensor is used to impute values of other sensors, they'll have a larger than normal discrepancies too. However, since this sensor is just one of them in the imputation, the discrepancies will be relatively small.

For instance, you could peek a very simple imputation by average: $$\hat x_k=\frac 1 {n-1} \sum_{i=1\\i\ne k}^n x_i$$ Suppose that sensor $j$ is faulty, then the discrepancy $$\hat x_j - x_j=\frac 1 {n-1} \sum_{i=1\\i\ne j}^n x_i$$ should much larger than usually, and also much larger than of a working sensor k: $\hat x_k-x_k$, becauase the faulty sensor comes with weight $1/n$ into calculation of imputed value $\hat x_k$

Answer 2

Multiple solutions :

1 ) Maybe a complicated one but that is really promising : The best way to exploit the correlations between time instants is probably to build a Multitask Gaussian Process that gives you a correlation matrix over time and features. By looking at the final correlation matrix, the outliers will be easily observable as they are clearly less correlated with the other series. In you example, a gap of intensity will be clearly distinguishable when the sensor starts to generate anomalies.

2) Just look at the auto-correlation... You can easily see a gap when the sensor starts to generate fake values.

3 ) An easy one that is really often used in the case of anomaly detection: you can look at the distribution of the data and approximate this distribution with a Mixture of Gaussian for example. You can then define the lower bound of the admissible values and categorize the rest as anomalies... In this case, it's important to have sufficient good data, so outliers are considered as exceptions...

4 ) If the data are highly correlated, it should be easy to generate some of the features with some others. You can build an autoencoder to predict the feature values. If the accuracy decreases drastically, you can be confident that this is an outlier.