Hi I have a large data set of objects, each containing a list of the same attributes. The data is arranged in a time series so that the value for an attribute for an object is indexed by its time. I want to model each attribute and perform some kind of anomaly detection. I also want to see which other attributes have similar patterns where the anomalies occur i.e. if they are correlated or dependent on each other. Once I have a model, I wish to use it on a test set to predict anomalies for an object before they occur. Any recommendations on how to approach this?
1
-
$\begingroup$ Can you provide a simple reproducible example ? $\endgroup$– forecasterJun 24, 2015 at 20:43
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
Use Intervention Detection procedures in conjunction with ARIMA . This will give you the time points where a pulse/level shift/seasonal pulse/local time trend was detectable/detected. Matching up these time periods will suggest commonalities.
answered Jun 24, 2015 at 20:22
-
$\begingroup$ Would you have any recommendations about how to model and analyze the possibility that the anomalies are "correlated or dependent on each other"? $\endgroup$– whuber ♦Jun 24, 2015 at 20:32
-
$\begingroup$ Some form of non-parametric test based upon the number of "hits or common occurrences" . A simple enumeration would be my start ( and perhaps finish ! ) as I don't really know the appropriiate/relevent literature here. $\endgroup$ Jun 24, 2015 at 20:35
$\begingroup$
$\endgroup$
There are multiple commercial products solving this sort of problem, sold as condition monitoring systems (for "Condition Based Maintenance" -- see Wikipedia), so you might try looking into what has been published about the underlying technologies.
Several approaches: (1) Use a modeling technique, and combine with a Sequential Probability Ratio Test (SPRT), as in
https://www.researchgate.net/publication/267969053 .
(2) Another method combining RDE clustering with fault detection & isolation could be used,
https://www.researchgate.net/publication/268690789
https://www.researchgate.net/publication/269038421
https://www.researchgate.net/publication/282650102
although that one is so data driven that it doesn't directly give you a model for predictive purposes. For that, you'd have to match up conditions at the time just before fault detection, as suggested in another answer.
