11
$\begingroup$

I need to implement anomaly detection on several time-series datasets. I've never done this before and was hoping for some advice. I'm very comfortable with python, so I would prefer the solution be implemented in it (most of my code is python for other parts of my work).

Description of the data: It's monthly time-series data that has only just begun to be collected in the past 2 years or so (i.e. only 24-36 time periods). Essentially, there are several metrics being monitored on a monthly basis for several clients.

time_period    client    metric    score
01-2013        client1   metric1   100
02-2013        client1   metric1   119
01-2013        client2   metric1   50
02-2013        client2   metric2   500
...

Here's what I'm thinking: pull data into a dataframe (pandas), then calculate a rolling 6 month average for each client / metric pair. If the current time period's value exceeds some threshold based on the 6-month avg., then raise flag. The problem seems rather simple. I just want to make sure I'm taking a solid approach.

Any advice to flesh this idea out a bit would be greatly appreciated. I know the question is a bit abstract, and I apologize for that.

Improve this question
$\endgroup$
2
  • $\begingroup$ I don't know the python way, but this question is full of ideas regarding the general approaches: stats.stackexchange.com/questions/26688/… $\endgroup$
    – rapaio
    Oct 23, 2014 at 6:27
  • $\begingroup$ pypi.org/project/anomaly-detection This is built in the library for anomaly detection in python which is similar to twitter anomaly detection. Since twitter anomaly detection code is in R language. Your problem is contextual anomaly. Auto.arima model too $\endgroup$
    – saravanan saminathan
    Jan 23, 2019 at 7:18

3 Answers 3

Reset to default
1
$\begingroup$

I think an approach similar to statistical process control, with control charts etc. might be useful here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'll read this. Is this method good for time series with small amounts of data (i.e. 24 months)? $\endgroup$
    – Eric Miller
    Oct 23, 2014 at 14:30
  • $\begingroup$ finished reading most of it. According to this method, I should calculate the 3rd standard deviation for the time series and graph a line on these limits. If a value ever exceeds these limits, then flag it. This is a method I had considered. $\endgroup$
    – Eric Miller
    Oct 23, 2014 at 14:44
1
$\begingroup$

There is plenty of options for anomaly detection, from a standard deviation using Pandas std deviation function, to a Bayesian method and many Machine learning methods in between like: clustering, SVM, Gaussian Process, Neural networks.

Take a look to this tutorial: https://www.datascience.com/blog/python-anomaly-detection

From a Bayesian perspective I recomend Facebook Prophet. It gives very advanced results without the need of being a Time series expert. It has the options for working on months, days etc, and "uncertainty intervals" help with anomalies.

Finally, I recomend this Uber blog about using Neural nets (LSTM) for anomaly detection, it has very goods insights: https://eng.uber.com/neural-networks/

Improve this answer
$\endgroup$
0
$\begingroup$

If you are willing to assume that your dataset is normally distributed, then you can estimate quantiles of this this distribution and see if it falls outside e.g 95%, 80%, etc quantile. I'm not too familiar with Python libraries but I'm sure there are already built functions for it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ There is a trend upward for most clients. I'm not sure you could call the data random. $\endgroup$
    – Eric Miller
    Oct 23, 2014 at 14:29
  • 2
    $\begingroup$ Then no in this case. I believe the method below me and yours would work well in this situation. I did something similar to this before: take a rolling X period moving average, subtract current metric value from the moving average. Find the standard deviation bounds (or use a subjective input if you do happen to know in this scenario) of these residual and anything above or below these bounds can be considered anomaly. This method would work well if a client suddenly sees a score increase. $\endgroup$
    – Kevin Pei
    Oct 23, 2014 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.