# Simple algorithm for online outlier detection of a generic time series II: Daily cycle within annual

I have several years of sensor data (temperature and relative humidity) that records every 1/2 hour. When the sensor dies, it often starts throwing bad data mixed in with good data before it dies completely. When it dies completely, it reports an error code ( like -100).

I've been trying to come up with an automated method to flag (and fill) bad data. I found @RobHyndman's Simple algorithm for online outlier detection of a generic time series R code and got it working but it assumes a simple seasonal pattern. If I were more conversant with STL, I might be able to figure out how to use it to replace outliers with expected values.

I've also found some of his (and his student's) other work on complex seasonality and wish I had the time now to absorb it all. In the meantime, my guess is that someone like him could probably add a line or two or a loop to that tsoutliers function and it would do just what I need.

I have posted the data to a google drive folder.

There are three files. LOESS2.csv has time and temperature (all that is needed for the posted question). The sensor starts going bad on 1/3/2013. I've added a snippet below of the first obviously bad values. If you want the good the bad and the ugly - which might suggest a couple alternative approaches - check out the other files.

1/2/2013 23:00  18.08
1/2/2013 23:30  18.02
1/3/2013 0:00   17.92
1/3/2013 0:30   -9.66
1/3/2013 1:00   -17.56
1/3/2013 1:30   17.61
1/3/2013 2:00   17.43
1/3/2013 2:30   17.26


A characteristic of the data that might help is each value is actually an average of more frequent measurements by a data logger. Perhaps as a result, they are fairly smooth in that they rarely "reverse direction" twice in a short period.

After a night of thinking about this, I am wondering if I should try to get at this using TBATS (or even VARS) instead of STL.

• Can you post sample data, so that people can play with different algorithms with a solution ? – forecaster May 11 '15 at 20:11
• @forecaster I'd be happy to. I thought the original questioner had done that so I was going to try to follow along. Got a tip or a link for the best way to post data? – M T May 11 '15 at 20:40
• btw, I just saw that the original question had been copied from a different stackexchange where adaptive filtering was mentioned as a possible solution. Is there an R package for that? – M T May 11 '15 at 20:41
• you could try dropbox or ge.tt or some other related sites, there are lots of questions in this forum has data link attached to it, you could look at those. – forecaster May 12 '15 at 16:00
• @forecaster, I put the data in a shared folder in google drive and added the link above. – M T May 12 '15 at 20:26

I'm not an expert on senor data. However, your data reminded me of click stream/ internet data. I recently stumbled upon twitters anomaly detection algorithm. I have not personally had great success with this method, but wanted to give a try on your data because of similarity of this data with data generated by click stream/tweets. I used annual cycle for seasonality (365*48 = 17520).

data <- read.csv("C:/Desktop/LOESS2.csv", header = TRUE)

data.outliers.annual <- AnomalyDetectionVec(data[,2],period = 17520, plot=TRUE,direction = "both")


Below is the plot from the above data:

Although this is not ideal, based on visual inspection this method does a decent job of detecting obvious outliers.

An alternative approach would be to look into state space methods to capture multi seasonality and them simultaneously detect outliers. I'll try to post if I find time.