Hi I have a large data set of objects, each containing a list of the same attributes. The data is arranged in a time series so that the value for an attribute for an object is indexed by its time. I want to model each attribute and perform some kind of anomaly detection. I also want to see which other attributes have similar patterns where the anomalies occur i.e. if they are correlated or dependent on each other. Once I have a model, I wish to use it on a test set to predict anomalies for an object before they occur. Any recommendations on how to approach this?
Use Intervention Detection procedures in conjunction with ARIMA . This will give you the time points where a pulse/level shift/seasonal pulse/local time trend was detectable/detected. Matching up these time periods will suggest commonalities.
There are multiple commercial products solving this sort of problem, sold as condition monitoring systems (for "Condition Based Maintenance" -- see Wikipedia), so you might try looking into what has been published about the underlying technologies.
Several approaches: (1) Use a modeling technique, and combine with a Sequential Probability Ratio Test (SPRT), as in
(2) Another method combining RDE clustering with fault detection & isolation could be used, https://www.researchgate.net/publication/268690789
https://www.researchgate.net/publication/269038421 https://www.researchgate.net/publication/282650102 although that one is so data driven that it doesn't directly give you a model for predictive purposes. For that, you'd have to match up conditions at the time just before fault detection, as suggested in another answer.