0
$\begingroup$

I want to detect anomalies in time series data. For me an anomaly is an abnormal value over a certain period of time.

Let's say in my time series i have usually a value around 100 with small variations, e.g. a smaller value like 90 for 3 seconds. An anomaly for me would be a value of 90 for 1 minute for example.

Of course I don't want to hardcode those thresholds. Is there an unsupervised anomaly detection method, that considers not only the value (like outlier detection), but also the time dimension?

$\endgroup$
0
$\begingroup$

What you need to do is review http://www.unc.edu/~jbhill/tsay.pdf it discusses the idea of ARIMA modelling and Intervention Detection ( e.g.level/step shifts and pulses and seasonal pulses and local time trends ). Care should be taken to make sure that the final model error variance is homogeneous/constant over time and that the model parameters are also invariant over time.

Some software implementations actually allow you to specify the minimum size of the step and the minimum number of values in the"new group" in order to reduce false positives. At a minimum the user should be able to specify the level of confidence and to also control the kinds of detected interventions that they wish to be enabled/disabled.

$\endgroup$
  • $\begingroup$ ok thanks, but maybe at first a basic question. What is the correct term of that kind of anomaly detection, where also the time dimension is considered? Is it "Intervention detection"? Are there also clustering approaches to do that? $\endgroup$ – MikeHuber Sep 10 '15 at 11:54
  • $\begingroup$ An anomaly detection speaks to the issue "what is the probability of an observation before it is observed" . A sequence of values over time can be characterized .. the question is is there an onset of an unusual value (pulse) or a sequence of "unusual values values around a new mean" (level/step shift) . It appears that you trying to automatically detect level/step shifts without hard-coding . In essence a segmentation is indeed a cluster BUT clustering methods as you may know them deal with multiple characteristics where you only have 1 characteristic (y) $\endgroup$ – IrishStat Sep 10 '15 at 12:06
  • $\begingroup$ In the last year or so we have implemented a scheme to detect whether or not a level shift meets a size requirement. We have always had the option to specify the minimum # within a group/set in order to call it a step/level shift. $\endgroup$ – IrishStat Feb 8 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.