I am tasked to develop an anomaly detection system for data organised in many 1D (can be more than 1D if I choose, but I think that will complicate the problem even more) daily time series. The series are largely unseasonal, but they may have trends. I started with two simplest implementations, namely exponential time-weighted moving average (i.e. Holt part of Holt-Winters method, since there is no seasonality) and a simple differenced series Δ(t)= Y(t) – Y(t-1) to detect sudden huge movements.
Ultimately, I want to implement an ensemble with a collection of algorithms, because different algorithms deal with different anomalies. (for e.g. immediately after an anomaly, SD is so big that my first algorithm is essentially useless; on the other hand, if there are two consecutive anomalies, second algorithm can’t detect the latter one) Also, I don’t know what kind of anomalies are more important to the end-user, so if I have an ensemble system allowing user to review the result, there can be a supervised learning algorithm to learn the relative weights of different algos.
My concerns are that 1) I am not sure my two simple algorithms are good/robust enough. I reviewed literature, and found more involved statistical/probabilistic methods such as Kalman filter, ARIMA (I actually implemented ARIMA, but iterating to optimal parameters is computationally expensive. Also, it seems fitting ARIMA in python is really painful - takes long time, and gives me loads of warnings like Fail to Converge, and sometimes it just fails outright saying MA/AR coefficients are not invertible. Because I have a lot of time series, it is not possible to visually inspect autocorrelation/partial autocorrelation graphs by eye. However, if you have solution to this problem, I am definitely willing to try) and machine learning algos (like clustering, k-neighbours, etc). I am wondering do are there any recommendations that may be relevant to my problem so I can make my approach more targeted?
2) While finding additive outliers is important, I am also interested implement change detection (e.g. detecting presence of ramp, mean change, variance change, etc.) A Google search did not yield too many promising results. Any suggestions on this area will be highly appreciated.
(using Python by the way)