15
$\begingroup$

I would like to setup up an algorithm for detecting an anomaly in time series, and I plan to use clustering for that.

  • Why should I use a distance matrix for clustering and not the raw time series data?,

  • For the detection of the anomaly, I will use density-based clustering, an algorithm as DBscan, so would that work in this case? Is there an online version for streaming data?

  • I would like to detect the anomaly before it happens, so , would using a trend detection algorithm (ARIMA) be a good choice?

$\endgroup$
  • $\begingroup$ It's correctly written DBSCAN. It is an abbreviation. I'm not sure what you are trying to do. Detect anomalies within a time series, or overall anomalous time series. $\endgroup$ – Anony-Mousse Apr 18 '12 at 16:07
  • $\begingroup$ Yes DBSCAN, exacte! What i'm trying to do, is an online detection anomaly in a time series dataset ! so! any request ? thanks regards $\endgroup$ – napsterockoeur Apr 19 '12 at 8:52
  • $\begingroup$ Online as in growing timeseries or as in additional series being added? Again, these are very different, and you need to be very clear on which you mean. $\endgroup$ – Anony-Mousse Apr 19 '12 at 18:43
  • $\begingroup$ I mean by online (stream), a growing times series coming from a sensor.. each one hour a set of data (vector) is recieved .. $\endgroup$ – napsterockoeur Apr 24 '12 at 8:20
12
$\begingroup$

Regarding your first question, I would recommend that you read this famous article (Clustering of Time Series Subsequences is Meaningless) before doing clustering on a time series. It is clearly written and illustrates many pitfalls that you want to avoid.

$\endgroup$
6
$\begingroup$

Anomaly detection or "Intervention Detection" has been championd by G.C.Tiao and others. To do science is to search for repeated patterns.To detect anomalies is to identify values that do not follow repeated patterns. We learn from Newton "Whoever knows the ways of Nature will more easily notice her deviations and, on the other hand, whoever knows her deviations will more accurately describe her ways". One learns the rules by observing when the current rules fail. Consider the time series 1,9,1,9,1,9,5,9 . To identify the anomaly one needs to have a pattern. The "5" is as much an anomaly as "14" would be . To identifY the pattern simply use ARIMA and in this case the "anomaly" becomes obvious. Try different software/approaches and see which one suggests an ARIMA model of order 1,0,0 with a coefficient of -1.0 . Use google/search procedures to find "automatic arima" or "automatic intervention detection". You may be disappointed by free stuff as it might be worth what you pay for it. Writing it yourself might be interesting if you have a heavy time series background and a couple of years to waste.There are serious limitations to distance based methods http://www3.ntu.edu.sg/SCE/pakdd2006/tutorial/chawla_tutorial_pakddslides.pdf

$\endgroup$
  • $\begingroup$ Thank you so much Sir IrishStat, I’m totally okay with u, that there a big limitations in distance based methods and i guess the other methods too, this is why I’m testing density base method, i saw a lot of articles speaking about times series anomaly detection, as nasa research, universities ..etc but small progress, for particular data problems And recently I found, a good free software for outliers detection : MOA of Weka ! Did you test it before? it’s an open source software, I’m trying to using it for developing and integrating my small detection anomaly algorithm, $\endgroup$ – napsterockoeur Apr 19 '12 at 9:02
  • $\begingroup$ oh : FYI :i'm treating a streaming data $\endgroup$ – napsterockoeur Apr 19 '12 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.