Top five classifiers to try first

Question

Besides obvious classifier characteristics like

• computational cost,
• expected data types of features/labels and
• suitability for certain sizes and dimensions of data sets,

what are the top five (or 10, 20?) classifiers to try first on a new data set one does not know much about yet (e.g. semantics and correlation of individual features)? Usually I try Naive Bayes, Nearest Neighbor, Decision Tree and SVM - though I have no good reason for this selection other than I know them and mostly understand how they work.

I guess one should choose classifiers which cover the most important general classification approaches. Which selection would you recommend, according to that criterion or for any other reason?

---

UPDATE: An alternative formulation for this question could be: "Which general approaches to classification exist and which specific methods cover the most important/popular/promising ones?"

Answer 1

Random Forest

Fast, robust, good accuracy, in most cases nothing to tune, requires no normalization, immune to collinearity, generates quite good error approximation and useful importance ranking as a side effect of training, trivially parallel, predicts in a blink of an eye.

Drawbacks: slower than trivial methods like kNN or NB, works best with equal classes, worse accuracy than SVM for problems desperately requiring kernel trick, is a hard black-box, does not make coffee.

Answer 2

Gaussian process classifier (not using the Laplace approximation), preferably with marginalisation rather than optimisation of the hyper-parameters. Why?

1. because they give a probabilistic classification
2. you can use a kernel function that allows you to operate directly on non-vectorial data and/or incorporate expert knowledge
3. they deal with the uncertainty in fitting the model properly, and you can propagate that uncertainty through to the decision making process
4. generally very good predictive performance.

Downsides

1. slow
2. requires a lot of memory
3. impractical for large scale problems.

First choice though would be regularised logistic regression or ridge regression [without feature selection] - for most problems, very simple algorithms work rather well and are more difficult to get wrong (in practice the differences in performance between algorithms is smaller than the differences in performance between the operator driving them).

Answer 3

By myself when you are approaching to a new data set you should start to watch to the whole problem. First of all get a distribution for categorical features and mean and standard deviations for each continuous feature. Then:

• Delete features with more than X% missing values;
• Delete categorical features when a particular value gets more then 90-95% of relative frequency;
• Delete continuous features with CV=std/mean<0.1;
• Get a parameter ranking, eg ANOVA for continuous and Chi-square for categorical;
• Get a significant subset of features;

Then I usually split the classification techniques in 2 sets: white box and black box technique. If you need to know 'how the classifier works' you should choose in the first set, eg Decision-Trees or Rules-based classifiers.

If you need to classify new records without building a model should should take a look to eager learner, eg KNN.

After that I think is better to have a threshold between accuracy and speed: Neural Network are a bit slower than SVM.

This is my top five classification technique:

1. Decision Tree;
2. Rule-based classifiers;
3. SMO (SVM);
4. Naive Bayes;
5. Neural Networks.