I often come across a classification problem - where we have 0/1 binary outcome and several features. And the main goal is build a classifier on training set.

Now given several choices of algorithms - Random forests, logistic regression, SVM, etc., is there a scientific approach one can apply to choose one among the above algorithms just based on the data attributes. By attributes I mean number of features in dataset, no. of categorical variables, how many levels in categorical variables, etc.

In other words, you have dataset and based on it you take a call which method suits best.

The reason I ask is that I currently apply different methods and choose one with the best accuracy on cross validation set. But I think there is a way to narrow down on methods just based on dataset features.

Would appreciate any thoughts on this.

Thanks in advance!


1 Answer 1


This scientific approach you ask for, I would call some reasonable rules of thumb, here's a list (please edit):

  • If a simple model is good enough, then stay with simple.
  • If you expect noisy data, use regularization and robust methods
  • If your data set is flat (more features than observations) your probably gonna need more regularization
  • If you expect unknown non-linear relationships, then use a non-linear learner(not standard logistic regression)
  • Pick a learner which is likely to fit your data structure well.
  • Resort to outer repeated cross-validation to confirm.
  • +10k observations, RF faster than SVM
  • SVM + One hot encoding may not work well for features with +~5 categories
  • randomFoerst will become slow +~15 categories (consider merging categories or use sklearn implementation or Rborist or extraTrees or xgboost(gradient boosting))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.