1
$\begingroup$

I have several years of sensor data (temperature and relative humidity) that records every 1/2 hour. When the sensor dies, it often starts throwing bad data mixed in with good data before it dies completely. When it dies completely, it reports an error code ( like -100).

I've been trying to come up with an automated method to flag (and fill) bad data. I found @RobHyndman's Simple algorithm for online outlier detection of a generic time series R code and got it working but it assumes a simple seasonal pattern. If I were more conversant with STL, I might be able to figure out how to use it to replace outliers with expected values.

I've also found some of his (and his student's) other work on complex seasonality and wish I had the time now to absorb it all. In the meantime, my guess is that someone like him could probably add a line or two or a loop to that tsoutliers function and it would do just what I need.

I have posted the data to a google drive folder.

There are three files. LOESS2.csv has time and temperature (all that is needed for the posted question). The sensor starts going bad on 1/3/2013. I've added a snippet below of the first obviously bad values. If you want the good the bad and the ugly - which might suggest a couple alternative approaches - check out the other files.

1/2/2013 23:00  18.08
1/2/2013 23:30  18.02
1/3/2013 0:00   17.92
1/3/2013 0:30   -9.66
1/3/2013 1:00   -17.56
1/3/2013 1:30   17.61
1/3/2013 2:00   17.43
1/3/2013 2:30   17.26

A characteristic of the data that might help is each value is actually an average of more frequent measurements by a data logger. Perhaps as a result, they are fairly smooth in that they rarely "reverse direction" twice in a short period.

After a night of thinking about this, I am wondering if I should try to get at this using TBATS (or even VARS) instead of STL.

$\endgroup$
  • 1
    $\begingroup$ Can you post sample data, so that people can play with different algorithms with a solution ? $\endgroup$ – forecaster May 11 '15 at 20:11
  • $\begingroup$ @forecaster I'd be happy to. I thought the original questioner had done that so I was going to try to follow along. Got a tip or a link for the best way to post data? $\endgroup$ – M T May 11 '15 at 20:40
  • $\begingroup$ btw, I just saw that the original question had been copied from a different stackexchange where adaptive filtering was mentioned as a possible solution. Is there an R package for that? $\endgroup$ – M T May 11 '15 at 20:41
  • $\begingroup$ you could try dropbox or ge.tt or some other related sites, there are lots of questions in this forum has data link attached to it, you could look at those. $\endgroup$ – forecaster May 12 '15 at 16:00
  • $\begingroup$ @forecaster, I put the data in a shared folder in google drive and added the link above. $\endgroup$ – M T May 12 '15 at 20:26
1
$\begingroup$

I'm not an expert on senor data. However, your data reminded me of click stream/ internet data. I recently stumbled upon twitters anomaly detection algorithm. I have not personally had great success with this method, but wanted to give a try on your data because of similarity of this data with data generated by click stream/tweets. I used annual cycle for seasonality (365*48 = 17520).

data <- read.csv("C:/Desktop/LOESS2.csv", header = TRUE)

data.outliers.annual <- AnomalyDetectionVec(data[,2],period = 17520, plot=TRUE,direction = "both")

Below is the plot from the above data:

enter image description here

Although this is not ideal, based on visual inspection this method does a decent job of detecting obvious outliers.

An alternative approach would be to look into state space methods to capture multi seasonality and them simultaneously detect outliers. I'll try to post if I find time.

Hope this was helpful

$\endgroup$
  • $\begingroup$ Interesting, and thanks for giving it a whirl. I guess the problem could be defined as "overdetection." There shouldn't be any anomalies in the first part of the dataset although perhaps that method has some parameters that could be tuned? In the meantime, I've been playing with the msts function in forecast ( something like "y <- msts(dr_x, seasonal.periods = c(48, 17520))). Once I get a handle on how to work with the output, I might get somewhere. $\endgroup$ – M T May 12 '15 at 21:41
  • $\begingroup$ When you say first part can you let me know which part? by the way seasonal period is c(48,365) $\endgroup$ – forecaster May 12 '15 at 21:52
  • $\begingroup$ +1 for effort. I've added some more description to the question. As for the (48, 365) I guess I need to RTFM. Actually, I did read the manual but it isn't really clear. Is that documented somewhere else? It just says "A vector of the seasonal periods of the msts" - no units mentioned anywhere I can find. So I got the impression that those periods should be in terms of the raw data's temporal resolution since most examples are in days. $\endgroup$ – M T May 13 '15 at 14:32
  • $\begingroup$ msts is not specified any where in the documentation, are you sure its the right package ? $\endgroup$ – forecaster May 13 '15 at 14:48
  • $\begingroup$ ?forecast::msts $\endgroup$ – M T May 13 '15 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.