3
$\begingroup$

I'm tackling the problem of Anomaly Detection in a dataset that's comprised of call counts to a call-centre. The data exhibits daily and weekly seasonality and is known to be over-dispersed.

Methodology:

The first step is to define "normal" behaviour, which is assumed to be the median from the previous n-weeks (median because it is robust to the presence of outliers in the data).

For example, say current (Monday 09:00AM) calls are 100, and the last n=5 Monday 09:00AM calls were [110,90,122,105,96], so the baseline "normal" calls for this Monday 09:00AM is the median=105. This gives a smoothed series which we define as "normal" (this is assumed sacrosanct).

Then I fit a negative-binomial model for previous n-weeks of this normal behaviour using MLE. Finally, I compute the likelihood of observed calls or more given the fitted model, and declare anomalies based off a threshold on the upper tail P(X>=observed) i.e. we are interested in unusually high # calls, ignoring # calls less than baseline.

Here is a plot of the algorithm in action:

note the red triangles

Now, i'm trying to evaluate this methodology by injecting artificial anomalies on the baseline series, which requires some knowledge of the residuals:

residual = observed - baseline (median of last n-weeks)

If I can simulate the residual generating process to inject known anomalies, knowing the correct p-values (labels), it would serve as an objective evaluation of this methodology in terms of accuracy of anomaly detection and quantification.

Questions:

  1. What form of the residuals should be simulated? in other words, is the choice of considering a raw residual (as above) correct?

  2. What should be the distribution of residuals? How do I simulate them?

  3. Any thoughts/comments/suggestions on the methodology itself?

Improve this question
$\endgroup$
3
  • $\begingroup$ As it stands, this question is not well-posed. What are you trying to evaluate about the model? How well it fits, or how robust it is to mis-specification, or how it handles outliers, or something else? $\endgroup$
    – JDL
    Aug 31, 2016 at 12:00
  • $\begingroup$ I'm trying to evaluate the model's accuracy in detecting anomalies by injecting known anomalous instances, so the question pertains to defining this anomalous process. $\endgroup$
    – tool.ish
    Aug 31, 2016 at 13:11
  • 1
    $\begingroup$ I think the problem you are having is that you are defining an outlier as "anything the model thinks is unlikely". By definition, a model will not often produce results it thinks are unlikely. As to p-values, if the model is "correct" they should be uniform on [0,1]. $\endgroup$
    – JDL
    Aug 31, 2016 at 13:16

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.