Besides obvious classifier characteristics like

  • computational cost,
  • expected data types of features/labels and
  • suitability for certain sizes and dimensions of data sets,

what are the top five (or 10, 20?) classifiers to try first on a new data set one does not know much about yet (e.g. semantics and correlation of individual features)? Usually I try Naive Bayes, Nearest Neighbor, Decision Tree and SVM - though I have no good reason for this selection other than I know them and mostly understand how they work.

I guess one should choose classifiers which cover the most important general classification approaches. Which selection would you recommend, according to that criterion or for any other reason?

UPDATE: An alternative formulation for this question could be: "Which general approaches to classification exist and which specific methods cover the most important/popular/promising ones?"

  • $\begingroup$ I think this is a duplicate of (stats.stackexchange.com/questions/3458/…). The only additional value of the question here could be "criterions for selecting classifiers" (which would make the question a very generic one). If it's a duplicate, vote for close, else vote for cw ;) $\endgroup$ – steffen Feb 25 '11 at 11:02
  • $\begingroup$ @steffen: Your referenced question is helpful, though, I think it's not a duplicate. Indeed my question is rather generic. I'm not looking for a solution to a particular problem but for general reasons why to use which learners - I'll update my question accordingly. $\endgroup$ – Oben Sonne Feb 25 '11 at 11:36

Random Forest

Fast, robust, good accuracy, in most cases nothing to tune, requires no normalization, immune to collinearity, generates quite good error approximation and useful importance ranking as a side effect of training, trivially parallel, predicts in a blink of an eye.

Drawbacks: slower than trivial methods like kNN or NB, works best with equal classes, worse accuracy than SVM for problems desperately requiring kernel trick, is a hard black-box, does not make coffee.

| cite | improve this answer | |
  • 1
    $\begingroup$ @mbq (+1) About class imbalance, we can still rely on stratified sampling during bagging. $\endgroup$ – chl Feb 25 '11 at 16:31
  • 3
    $\begingroup$ @mbq, doesn't make coffee? That's a deal-breaker right there. $\endgroup$ – cardinal Feb 25 '11 at 18:06
  • $\begingroup$ Thanks for the hint to Random Forests. But would you try only them? What if you're not happy with the results? Which classifier you would try else? Or, what would you answer if someone asks: "Why didn't you try other methods?" $\endgroup$ – Oben Sonne Mar 1 '11 at 20:49
  • $\begingroup$ @Oben Well, I understood you are making a kind of one-classifier-per-answer pool. $\endgroup$ – user88 Mar 1 '11 at 20:55
  • $\begingroup$ @mbq: Not really, but it turns out to be such a pool. Probably I did not make myself clear enough in the question. Actually I wanted to know which set of classifiers one should try first, to cover different general classification methods (with different strengths and weaknesses). I always ask myself if I shouldn't try more classifiers. Knowing that the ones I tried already represent the most typical/promising approaches would help here. But for that I need to know for which set of classifiers this is true. (I'm far from being a stats expert, so let me know if my mind is a bit twisted here) $\endgroup$ – Oben Sonne Mar 2 '11 at 7:07

Gaussian process classifier (not using the Laplace approximation), preferably with marginalisation rather than optimisation of the hyper-parameters. Why?

  1. because they give a probabilistic classification
  2. you can use a kernel function that allows you to operate directly on non-vectorial data and/or incorporate expert knowledge
  3. they deal with the uncertainty in fitting the model properly, and you can propagate that uncertainty through to the decision making process
  4. generally very good predictive performance.


  1. slow
  2. requires a lot of memory
  3. impractical for large scale problems.

First choice though would be regularised logistic regression or ridge regression [without feature selection] - for most problems, very simple algorithms work rather well and are more difficult to get wrong (in practice the differences in performance between algorithms is smaller than the differences in performance between the operator driving them).


By myself when you are approaching to a new data set you should start to watch to the whole problem. First of all get a distribution for categorical features and mean and standard deviations for each continuous feature. Then:

  • Delete features with more than X% missing values;
  • Delete categorical features when a particular value gets more then 90-95% of relative frequency;
  • Delete continuous features with CV=std/mean<0.1;
  • Get a parameter ranking, eg ANOVA for continuous and Chi-square for categorical;
  • Get a significant subset of features;

Then I usually split the classification techniques in 2 sets: white box and black box technique. If you need to know 'how the classifier works' you should choose in the first set, eg Decision-Trees or Rules-based classifiers.

If you need to classify new records without building a model should should take a look to eager learner, eg KNN.

After that I think is better to have a threshold between accuracy and speed: Neural Network are a bit slower than SVM.

This is my top five classification technique:

  1. Decision Tree;
  2. Rule-based classifiers;
  3. SMO (SVM);
  4. Naive Bayes;
  5. Neural Networks.
| cite | improve this answer | |
  • 4
    $\begingroup$ -1 Absolutely incorrect workflow for large p small n, FS overfitting is guaranteed. $\endgroup$ – user88 Feb 25 '11 at 14:51
  • 1
    $\begingroup$ isn't kNN a lazy learner rather than an eager one (as you don't actually do anything until you really have to when a pattern to classify comes along)? Any pre-processing you do before applying the classifier is likely to have a larger effect on performance than the difference between classifiers; feature selection is especially difficult (easily leads to over-fitting), and methods like the SVM with regularisation usually perform better without feature selection. I certainly wouldn't recommend neural networks, far too many potential pitfalls. $\endgroup$ – Dikran Marsupial Feb 25 '11 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.