What determines what classification algorithm you should use for a certain classification problem? e.g. If there is >5 features or you only have 1000 training examples, or there is multiple class's or it's binary classification use; logistic regression, SVM, ANN etc.


This flow chart is limited, but boils things down well for people new to the field. Beyond that, the dominating factors tend to be

  • What's available in my libraries?
  • What am I comfortable with?
  • Which models have assumptions that match my data?

That last one is quite a deep question, and the ability to answer it effectively only comes with a lot of study. If you've got a good grounding in linear algebra and multivariable calculus, a general ML book like Machine Learning: A Probabilistic Perspective would be a good place to start.

  • $\begingroup$ Thanks Andy for the reply. When it comes down to choosing does the amount of data you have or how many features come into play at all with regards to choosing an appropriate algorithm? $\endgroup$ – frog1944 Apr 6 '15 at 9:50
  • $\begingroup$ Both sample size and number of features are important, yes. $\endgroup$ – Andy Jones Apr 6 '15 at 10:27
  • 1
    $\begingroup$ Is there certain algorithms that are optimized for more or less features? Such as logistic regression for 4 features, but neural networks for >10 features etc. $\endgroup$ – frog1944 Apr 6 '15 at 23:30
  • $\begingroup$ Not so much 'optimized for' as 'suited for', but yes. k-means for example has problems with lots of features due to the curse of dimensionality. I can't think of any kind of model that's explicitly bad on small numbers of features, but there are lots that would likely be 'overkill'. $\endgroup$ – Andy Jones Apr 7 '15 at 20:10

It is pretty reasonable to start with a simple linear model and then work from there. Analysis of results will reveal the shortcomings of your features and/or algorithm. A significant advantage that you can get started quickly which is rather important in many practical applications.

The second step would be to start to use domain knowledge. Good analysis should result in a set of features that even a liner classifier might work satisfactory enough.

  • $\begingroup$ What linear model would you recommend? $\endgroup$ – frog1944 Apr 6 '15 at 23:30
  • $\begingroup$ @frog1944 Just start with a plain logistic regression or with linear SVM. The learning algorithms for linear models are blazingly fast, something that is important when playing with large dataset. $\endgroup$ – Vladislavs Dovgalecs Apr 7 '15 at 1:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.