Hi I have a large data set of objects, each containing a list of the same attributes. The data is arranged in a time series so that the value for an attribute for an object is indexed by its time. I want to model each attribute and perform some kind of anomaly detection. I also want to see which other attributes have similar patterns where the anomalies occur i.e. if they are correlated or dependent on each other. Once I have a model, I wish to use it on a test set to predict anomalies for an object before they occur. Any recommendations on how to approach this?

  • $\begingroup$ Can you provide a simple reproducible example ? $\endgroup$ – forecaster Jun 24 '15 at 20:43

Use Intervention Detection procedures in conjunction with ARIMA . This will give you the time points where a pulse/level shift/seasonal pulse/local time trend was detectable/detected. Matching up these time periods will suggest commonalities.

  • $\begingroup$ Would you have any recommendations about how to model and analyze the possibility that the anomalies are "correlated or dependent on each other"? $\endgroup$ – whuber Jun 24 '15 at 20:32
  • $\begingroup$ Some form of non-parametric test based upon the number of "hits or common occurrences" . A simple enumeration would be my start ( and perhaps finish ! ) as I don't really know the appropriiate/relevent literature here. $\endgroup$ – IrishStat Jun 24 '15 at 20:35

There are multiple commercial products solving this sort of problem, sold as condition monitoring systems (for "Condition Based Maintenance" -- see Wikipedia), so you might try looking into what has been published about the underlying technologies. Several approaches: (1) Use a modeling technique, and combine with a Sequential Probability Ratio Test (SPRT), as in https://www.researchgate.net/publication/267969053 .
(2) Another method combining RDE clustering with fault detection & isolation could be used, https://www.researchgate.net/publication/268690789
https://www.researchgate.net/publication/269038421 https://www.researchgate.net/publication/282650102 although that one is so data driven that it doesn't directly give you a model for predictive purposes. For that, you'd have to match up conditions at the time just before fault detection, as suggested in another answer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.