I'm building a predictive model with a combination of numeric, binary, and categorical variables. The outcome is binary.

For methods like SVM, I have read on stack exchange that the categorical features must be converted to binary variables, i.e for a variable of "hair color" that can take on multiple levels (e.g red hair, blonde hair, black hair, or brown hair) we create a binary variable for each level (Hair red? y/n, hair blonde? y/n, etc)

However, I'm concerned about linear dependence between the variables when I do this. In the hair color example, assuming everyone has hair color of either red, blonde, black or brown and that people only have 1 type of hair color, if I create 4 binary variables for hair color, then the sum of the 4 variables is always 1. In other words, I can write v1 + v2 + v2 + v4 = 1, which means there is a linear dependence between the binary variables I have introduced to replace the multi level categorical variable.

If I create only k-1 variables then I don't have that issue.

I understand that multi-collinearity is a problem when fitting a regression model with continuous data because the inverse of the covariance matrix is unstable. Is the issue of linear dependence/multi-collinearity a concern with SVM and Random Forest when using binary/categorical data?


With random forests? No. They don't look at linear combinations of features, so this isn't a problem, and actually if you're doing this one-hot feature representation where, say, you don't include brown hair (because it's equivalent to not-red, not-blond, not-black), then individual trees will have a hard time asking about brown hair because they need to have three randomly included features to do that. If you include a brown hair feature, it can. Many random forest packages, though definitely not all, allow "real" categorical inputs, and so you don't have to do the one-hot representation.

With SVMs? Generally no. A distance-based kernel, e.g. the RBF kernel, definitely does not have this issue. Linear or polynomial kernels or so on do use linear combinations in that way, and so if you don't regularize, you can run into the same multicollinearity problems as you do with OLS. But you should almost never not regularize an SVM; the standard $L_2$ regularization is equivalent to doing ridge regression, which handles most of the issues with multicollinearity, as noted in this thread linked to by @DJohnson above.

  • $\begingroup$ But using k binary variables to replace a k-level factor variable ina logistic regression is still problematic? $\endgroup$ – Nirvan Sengupta Nov 28 '15 at 18:31
  • 1
    $\begingroup$ Logistic regression is similar to SVMs here in that it's usually going to be $L_2$-regularized. If you only care about predictions, it's fine; if you care about inference for the coefficients, it can be a problem, in the same way it's a problem for ridge regression. $\endgroup$ – Danica Nov 28 '15 at 22:18
  • $\begingroup$ @Dougal Encoding a categorical variable with k levels to k-1 binaries is recommended as per this post stats.stackexchange.com/questions/186031/…. But for random forests and SVM you say encoding into k binary variables is good. Now i am confused. $\endgroup$ – prashanth Feb 25 '16 at 6:41
  • $\begingroup$ @prashanth That other post is talking only about linear regression-type models, where rank-deficiency is an issue. For random forests, it's not an issue, and sometimes advantageous to include all k. For SVMs with distance-based kernels, e.g. Gaussian RBF, it's preferable to include all of them because it makes the distances between different categories consistent; otherwise your "default" category is distance $1$ to everything, but all the others are $\sqrt 2$ apart. For linear-kernel SVMs, it can possibly be a problem to have all k, but probably not. $\endgroup$ – Danica Feb 25 '16 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.