21
$\begingroup$

I would like to setup up an algorithm for detecting an anomaly in time series, and I plan to use clustering for that.

  • Why should I use a distance matrix for clustering and not the raw time series data?,

  • For the detection of the anomaly, I will use density-based clustering, an algorithm as DBscan, so would that work in this case? Is there an online version for streaming data?

  • I would like to detect the anomaly before it happens, so , would using a trend detection algorithm (ARIMA) be a good choice?

Improve this question
$\endgroup$
4
  • $\begingroup$ It's correctly written DBSCAN. It is an abbreviation. I'm not sure what you are trying to do. Detect anomalies within a time series, or overall anomalous time series. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Apr 18, 2012 at 16:07
  • $\begingroup$ Yes DBSCAN, exacte! What i'm trying to do, is an online detection anomaly in a time series dataset ! so! any request ? thanks regards $\endgroup$
    – napsterockoeur
    Commented Apr 19, 2012 at 8:52
  • $\begingroup$ Online as in growing timeseries or as in additional series being added? Again, these are very different, and you need to be very clear on which you mean. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Apr 19, 2012 at 18:43
  • $\begingroup$ I mean by online (stream), a growing times series coming from a sensor.. each one hour a set of data (vector) is recieved .. $\endgroup$
    – napsterockoeur
    Commented Apr 24, 2012 at 8:20

3 Answers 3

Reset to default
12
$\begingroup$

Regarding your first question, I would recommend that you read this famous article (Clustering of Time Series Subsequences is Meaningless) before doing clustering on a time series. It is clearly written and illustrates many pitfalls that you want to avoid.

Improve this answer
$\endgroup$
0
6
$\begingroup$

Anomaly detection or "Intervention Detection" has been championd by G.C.Tiao and others. To do science is to search for repeated patterns.To detect anomalies is to identify values that do not follow repeated patterns. We learn from Newton "Whoever knows the ways of Nature will more easily notice her deviations and, on the other hand, whoever knows her deviations will more accurately describe her ways". One learns the rules by observing when the current rules fail. Consider the time series 1,9,1,9,1,9,5,9 . To identify the anomaly one needs to have a pattern. The "5" is as much an anomaly as "14" would be . To identifY the pattern simply use ARIMA and in this case the "anomaly" becomes obvious. Try different software/approaches and see which one suggests an ARIMA model of order 1,0,0 with a coefficient of -1.0 . Use google/search procedures to find "automatic arima" or "automatic intervention detection". You may be disappointed by free stuff as it might be worth what you pay for it. Writing it yourself might be interesting if you have a heavy time series background and a couple of years to waste.There are serious limitations to distance based methods http://www3.ntu.edu.sg/SCE/pakdd2006/tutorial/chawla_tutorial_pakddslides.pdf

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you so much Sir IrishStat, I’m totally okay with u, that there a big limitations in distance based methods and i guess the other methods too, this is why I’m testing density base method, i saw a lot of articles speaking about times series anomaly detection, as nasa research, universities ..etc but small progress, for particular data problems And recently I found, a good free software for outliers detection : MOA of Weka ! Did you test it before? it’s an open source software, I’m trying to using it for developing and integrating my small detection anomaly algorithm, $\endgroup$
    – napsterockoeur
    Commented Apr 19, 2012 at 9:02
  • $\begingroup$ oh : FYI :i'm treating a streaming data $\endgroup$
    – napsterockoeur
    Commented Apr 19, 2012 at 9:11
1
$\begingroup$

For time series anomaly detection there can be multiple approaches. As you have said, if you are using ARIMA as the model, you can use MAPE or SMAPE as the error metric and use a confidence threshold using it. Anything falling beyond the CI band can be an anomaly. Similarly you can go for DBSCAN or statistical profiling based approaches. For more information you can go through these links: https://towardsdatascience.com/effective-approaches-for-time-series-anomaly-detection-9485b40077f1 https://www.aditya-bhattacharya.com/?p=72&page=4

Let me know if it helps. :)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.