• one classifier per answer
  • vote up if you agree
  • downvote/remove duplicates.
  • put your application in the comment

10 Answers 10


Random forest

  • easily captures complicated structure/nonlinear relationship
  • invariant to variables' scale
  • no need to create dummy variables for categorical predictors
  • variable selection is not much needed
  • relatively hard to overfit
  • $\begingroup$ Aptamer active motif selection, forest ground humidity forecasting, digit OCR, multispectral satellite image analysis, musical information retrieval, chemometry... $\endgroup$ – user88 Jul 21 '10 at 15:58

Logistic Regression:

  • fast and perform well on most datasets
  • almost no parameters to tune
  • handles both discrete/continuous features
  • model is easily interpretable
  • (not really restricted to binary classifications)
  • $\begingroup$ Maybe no parameters to tune, but one has to really work with continuous variables (transformations, splines, etc) to induce non linearity. $\endgroup$ – B_Miner Nov 17 '11 at 0:01

Support vector machine

  • $\begingroup$ There isn't anything really special about the SVM, other than it forces the user to think about regularisation. For most practical problems [kernel] ridge regression works just as well. $\endgroup$ – Dikran Marsupial Mar 30 '11 at 11:36
  • 2
    $\begingroup$ @dikran i think SVM is a great classifier because it is sparse and robust to outliers -- this is not true for Logistic Regression! and thats why SVM is state-of-the-art classifier. Only issue which may be a problem is -- time complexity -- but i think its ok. $\endgroup$ – suncoolsu Mar 30 '11 at 12:17
  • $\begingroup$ @suncoolsu If you want sparsity, you get more sparsity from regularised logistic regression with LASSO than you do with the SVM. The sparsity of the SVM is a by-product of the loss function, so you don't get as much as you do with an algorithm where sparsity is a design goal. Also often with the optimal value of the hyper-parameter (e.g. chosen via cross-validation) most of the sparsity of the SVM dissapears. SVM is no more robust to outliers than regularised logistic regression - it is mostly the regularisation that matters, not the hinge loss. $\endgroup$ – Dikran Marsupial Mar 30 '11 at 12:22
  • $\begingroup$ @Dikran -- my point exactly -- some kind of penalization is important. You can get that using Priors, adding Penalty, etc. $\endgroup$ – suncoolsu Mar 30 '11 at 12:29
  • 1
    $\begingroup$ @suncoolsu In which case, the SVM isn't a great classifier, it is just one amongst many regularised classifiers, such as ridge regression, regularised logistic regression, Gaussian Processes. The main benefir of the SVM is its appeal from computational learning theory. In practice, other considerations are more important, such as whether you need probabilistic classifier, where other loss functions are likely to be superior. IMHO, there is too much attention given to the SVM, rather than the wider family of kernel methods. $\endgroup$ – Dikran Marsupial Mar 30 '11 at 12:37

Regularized discriminant for supervised problems with noisy data

  1. Computationally efficient
  2. Robust to noise and outliers in data
  3. Both linear discriminant (LD) and quadratic discriminant (QD) classifiers can can be obtained from the same implementation setting the regularization parameters '[lambda, r]' to '[1 0]' for LD classifier and '[0 0]' for QD classifier - very useful for reference purposes.
  4. Model is easy to interpret and export
  5. Works well for sparse and 'wide' data sets where class covariance matrices may not be well defined.
  6. An estimate of posterior class probability can be estimated for each sample by applying the softmax function to the discriminant values for each class.

Link to original 1989 paper by Friedman et al here. Also, there very good explanation by Kuncheva in her book "Combining pattern classifiers".


Gradient Boosted Trees.

  • At least as accurate as RF on a lot of applications
  • Incorporates missing values seamlessly
  • Var importance (like RF probably biased in favor of continuous and many level nominal)
  • Partial dependency plots
  • GBM versus randomForest in R : handles MUCH larger datasets

Gaussian Process classifier - it gives probabilistic predictions (which is useful when your operational relative class frequencies differ from those in your training set, or equivalenty your false-positive/false-negative costs are unknown or variable). It also provides an inidcation of the uncertainty in model predictions due to the uncertainty in "estimating the model" from a finite dataset. The co-variance function is equivalent to the kernel function in an SVM, so it can also operate directly on non-vectorial data (e.g. strings or graphs etc). The mathematical framework is also neat (but don't use the Laplace approximation). Automated model selection via maximising marginal likelihood.

Essentially combines good features of logistic regression and SVM.

  • $\begingroup$ Is there are R package that you recommend that implements this? What is your preferred implementation for this method? Thanks! $\endgroup$ – julieth Sep 26 '12 at 17:02
  • $\begingroup$ I'm afraid I am a MATLAB user (I use the GPML package gaussianprocess.org/gpml/code/matlab/doc), so I can't advise about R implementations, but you may find something suitable here gaussianprocess.org/#code . If R doesn't have a decent package for GPs, someone needs to write one! $\endgroup$ – Dikran Marsupial Sep 26 '12 at 17:04
  • $\begingroup$ Ok thanks. Does this methodolgy allow one to select "important variables, such as in the variable importance of random forests or recursive feature elimination with SVMs? $\endgroup$ – julieth Sep 26 '12 at 18:01
  • $\begingroup$ Yes, you can use an "Automatic Relevance Determination" covariance function, and choose the hyper-parameters by maximising the Bayesian evidence for the model (although this can run into the same sort of over-fitting problems you get with SVMS, so often the model performs better without feature selection). $\endgroup$ – Dikran Marsupial Sep 26 '12 at 18:05

L1-regularized logistic regression.

  • It is computationally fast.
  • It has an intuitive interpretation.
  • It has only one easily understandable hyperparameter that can be automatically tuned by cross-validation, which often is a good way to go.
  • Its coefficients are piecewise linear and their relation to the hyperparameter is instantly and easily visible in a simple plot.
  • It is one of the less dubious methods for variable selection.
  • Also it has a really cool name.



Naive Bayes and Random Naive Bays

  • 2
    $\begingroup$ Can you give a description a problem where RNB gave you good results? $\endgroup$ – Łukasz Lew Jul 21 '10 at 10:57
  • $\begingroup$ No ;-) This was only to revive the pool. $\endgroup$ – user88 Jul 21 '10 at 11:05

K-means clustering for unsupervised learning.

  • $\begingroup$ The question specifically asks for a classifier. $\endgroup$ – Prometheus Apr 11 '16 at 11:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.