1
$\begingroup$

I've proposed using an anomaly detection algorithm in a project.

The algorithm would consist of choosing some features we think might be indicative of anomalous examples. Then using a training set of anomalous and non-anomalous examples to fit parameters to a Gaussian distribution. We would then use these parameters to create a probability function p where p(x) < epsilon for some epsilon would indicate an anomaly.

How would I answer this challenge?

“We love this concept, but how does this differ from us just doing some statistics on a set of results?”

By just doing some statistics, I assume the challenger means detecting the anomalies manually in old data and coming up with a set of conditions that would indicate anomalies in future data.

$\endgroup$
1
  • 3
    $\begingroup$ This question in its current form appears to be unanswerable. Could you tell us what detection algorithm is proposed and--more importantly--exactly what your challenger might mean by "just doing some statistics"? $\endgroup$
    – whuber
    Jul 24, 2012 at 15:57

1 Answer 1

2
$\begingroup$

What your challenger was perhaps trying to communicate is that the problem you are posing is a very standard problem in statistics and lots is known about how to solve it and using one of the standard solutions may be easier and better. Rephrasing your problem in statistical lingo:

  • Your "training set" is a "sample".
  • Each of your "examples" is an "observation".
  • The variable indicating whether or not an anomaly exists is your "dependent" variable.
  • The variables the features of the examples are your "independent" variables (also known as predictors, covariates, etc.)

Lots of different algorithms have been developed for this problem, including regression, CART, discriminant analysis and so on. A wonderful book which gives detail about the various methods that have been developed is The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Second Edition) by Trevor Hastie, Robert Tibshirani and Jerome Friedman (2009)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.