What is the best way to automatically select features for anomaly detection?

I normally treat Anomaly Detection as an algorithm where the features are selected by human experts: what matters is the output range (as in "abnormal input - abnormal output") so even with many features you can come up with a much smaller subset by combining the features.

However, assuming that in general case a feature list can be huge, perhaps an automated learning is sometimes preferable. As far as I can see, there are some attempts:

  • "Automated feature selection for Anomaly Detection" (pdf) which generalizes Support Vector Data Description
  • "A Fast Host-Based Intrusion Detection System Using Rough Set Theory" (no pdf available?) which, I guess, uses Rough Set Theory
  • "Learning Rules for Anomaly Detection of Hostile Network Traffic" (pdf, video) which uses statistical approach

So now I wonder if anyone can tell - assuming anomaly detection and a really big (hundreds?) feature set:

  1. Do those huge feature sets make sense at all? Shouldn't we just reduce the feature set upto, say, a few dozens and that's it?
  2. If huge feature sets do make sense, which one of the approaches above would give better predictions, and why? Is there anything not listed which is much better?
  3. Why should they give better results comparing to, say, dimensionality reduction or feature construction via clustering/ranking/etc?
  • $\begingroup$ Your link didn't bring up any specific question for me. Can you give a brief description of the problem? What's the objective? Is it a supervised or unsupervised learning problem? $\endgroup$
    – AdamO
    Feb 23, 2012 at 0:11
  • $\begingroup$ The question was from ML.SE which is closed now - apparently the admins hadn't merged in all the questions. I edited the text now to explicitly state the problem! $\endgroup$
    – andreister
    Feb 23, 2012 at 6:30

1 Answer 1


One practical approach (in case of supervised learning at least) is to include all possibly relevant features and use a (generalized) linear model (logistic regression, linear svm etc.) with regularization (L1 and/or L2). There are open source tools (e.g. Vowpal Wabbit) that can deal with trillions of example/feature combinations for these types of models so scalability is not an issue (besides, one can always use sub-sampling). The regularization helps to deal with feature selection.

  • 1
    $\begingroup$ But how to select features in unsupervised settings (without needing to use linear models, etc., to come up with important features)? One way I can think of is using PCA to retain some variance and reducing the data size. But again, reducing data in anomaly detection problem seems dangerous as you might end up missing actual outliers that you wanted to predict. Hence the confusion. $\endgroup$
    – exAres
    Nov 18, 2015 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.