3
$\begingroup$

I am working with time series data of sensor measurements. I have nine sensors that are in the same ballpark location recording the same data every 10 minutes. The sensors are setup such that the readings should vary slightly given the setup of the experiment.

Some sensors experience power or other issues that causes them to record bad data. I would like to use the other data recordings and the trends associated, to be able to best detect outliers / anomalies from a given sensor. Ultimately I would like to label that data NaN. An example of the data I am looking at is provided here: Time Series from multiple ground sensors

As is evident, one of the sensors is having an issue and is fixed. Is there a preferred method for detecting outliers like this? Is it acceptable to just do an IQR using all the data? I am working in Python with Pandas for my data analysis.

$\endgroup$
  • 1
    $\begingroup$ it is possible to detect outliers when modelling causal series . The process is called Intervention Detection pdfs.semanticscholar.org/09c4/… and is applicable to not only sarma models but sarmaX models (also known as Transfer Functions ) autobox.com/pdfs/SARMAX.pdf where the I series are the Intervention Series waiting to be discovered. If you post an example including the Y and the X series in a csv file I will try and help further. $\endgroup$ – IrishStat May 6 '19 at 15:05
  • 1
    $\begingroup$ If this plot is characteristic of the problem (in the sense of the relative magnitude of the deviation and insofar as only one or two sensors might be affected at once), just about any univariate outlier detection method, applied independently over time, would be a good start. Do you need to detect the outlying fugues in real time or after the fact? What purpose(s) would the relabeling as NA ultimately serve? $\endgroup$ – whuber May 6 '19 at 15:08
  • 1
    $\begingroup$ @whuber After the fact is fine. Relabeling the obvious outliers, which hopefully represent bad sensor data, to NA allows other algorithms like change detection to ignore the NA values. Marking as NA also allows for tallying statistics on the bad data being recorded. $\endgroup$ – Doug May 9 '19 at 12:14
1
$\begingroup$

I suggest a simple "voting" mechanism: remove top and bottom n readings from each observation. This is often used in scoring of sporting events, such as diving where out of 10 judges top and bottom 2 scores are removed from calculations.

This is not the most optimal approach, of course, especially if it's rare that the sensor is faulty. However, it's very simple and robust. Even if you decide to employ more sophisticated approach, I'd still keep this one as a benchmark.

Another approach, if to constantly impute the measurement of each sensor based on the readings of other sensors. When the imputed value is different enough from an actual measurement, it indicates the problem. Of course, you'd have to take into account, that when the faulty sensor is used to impute values of other sensors, they'll have a larger than normal discrepancies too. However, since this sensor is just one of them in the imputation, the discrepancies will be relatively small.

For instance, you could peek a very simple imputation by average: $$\hat x_k=\frac 1 {n-1} \sum_{i=1\\i\ne k}^n x_i$$ Suppose that sensor $j$ is faulty, then the discrepancy $$\hat x_j - x_j=\frac 1 {n-1} \sum_{i=1\\i\ne j}^n x_i$$ should much larger than usually, and also much larger than of a working sensor k: $\hat x_k-x_k$, becauase the faulty sensor comes with weight $1/n$ into calculation of imputed value $\hat x_k$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Imputing the measurement of each sensor and checking discrepancy is an interesting idea. I don't think I can do the voting system because each sensor represents a different condition for the experiment, so they shouldn't all be getting the same measurement, but should be seeing similar trends and ranges... if that makes sense. $\endgroup$ – Doug May 9 '19 at 12:18
  • 1
    $\begingroup$ @Doug you can make imputing as simple as a multiple regression of a sensor on other sensors, no need to make this complicated $\endgroup$ – Aksakal May 9 '19 at 12:27
  • $\begingroup$ Marking as answer because I think the approach for now is exactly as @Aksakal puts it: multiple regression of a sensor on other sensors. Thanks for all the help. $\endgroup$ – Doug Aug 13 '19 at 15:27
1
$\begingroup$

Multiple solutions :

1 ) Maybe a complicated one but that is really promising : The best way to exploit the correlations between time instants is probably to build a Multitask Gaussian Process that gives you a correlation matrix over time and features. By looking at the final correlation matrix, the outliers will be easily observable as they are clearly less correlated with the other series. In you example, a gap of intensity will be clearly distinguishable when the sensor starts to generate anomalies.

2) Just look at the auto-correlation... You can easily see a gap when the sensor starts to generate fake values.

3 ) An easy one that is really often used in the case of anomaly detection: you can look at the distribution of the data and approximate this distribution with a Mixture of Gaussian for example. You can then define the lower bound of the admissible values and categorize the rest as anomalies... In this case, it's important to have sufficient good data, so outliers are considered as exceptions...

4 ) If the data are highly correlated, it should be easy to generate some of the features with some others. You can build an autoencoder to predict the feature values. If the accuracy decreases drastically, you can be confident that this is an outlier.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the multiple approaches. I'll have to do a bit of research on the methods you provided before prototyping and seeing if this ends up being a good solution. Cheers. $\endgroup$ – Doug May 9 '19 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.