Choice of classification method based on dataset attributes

Question

I often come across a classification problem - where we have 0/1 binary outcome and several features. And the main goal is build a classifier on training set.

Now given several choices of algorithms - Random forests, logistic regression, SVM, etc., is there a scientific approach one can apply to choose one among the above algorithms just based on the data attributes. By attributes I mean number of features in dataset, no. of categorical variables, how many levels in categorical variables, etc.

In other words, you have dataset and based on it you take a call which method suits best.

The reason I ask is that I currently apply different methods and choose one with the best accuracy on cross validation set. But I think there is a way to narrow down on methods just based on dataset features.

Would appreciate any thoughts on this.

Thanks in advance!

Answer 1

This scientific approach you ask for, I would call some reasonable rules of thumb, here's a list (please edit):

• If a simple model is good enough, then stay with simple.
• If you expect noisy data, use regularization and robust methods
• If your data set is flat (more features than observations) your probably gonna need more regularization
• If you expect unknown non-linear relationships, then use a non-linear learner(not standard logistic regression)
• Pick a learner which is likely to fit your data structure well.
• Resort to outer repeated cross-validation to confirm.
• +10k observations, RF faster than SVM
• SVM + One hot encoding may not work well for features with +~5 categories
• randomFoerst will become slow +~15 categories (consider merging categories or use sklearn implementation or Rborist or extraTrees or xgboost(gradient boosting))