# Should I convert a categorical variable with k levels to (k-1) or k binary variables?

I'm building a predictive model with a combination of numeric, binary, and categorical variables. The outcome is binary.

For methods like SVM, I have read on stack exchange that the categorical features must be converted to binary variables, i.e for a variable of "hair color" that can take on multiple levels (e.g red hair, blonde hair, black hair, or brown hair) we create a binary variable for each level (Hair red? y/n, hair blonde? y/n, etc)

However, I'm concerned about linear dependence between the variables when I do this. In the hair color example, assuming everyone has hair color of either red, blonde, black or brown and that people only have 1 type of hair color, if I create 4 binary variables for hair color, then the sum of the 4 variables is always 1. In other words, I can write v1 + v2 + v2 + v4 = 1, which means there is a linear dependence between the binary variables I have introduced to replace the multi level categorical variable.

If I create only k-1 variables then I don't have that issue.

I understand that multi-collinearity is a problem when fitting a regression model with continuous data because the inverse of the covariance matrix is unstable. Is the issue of linear dependence/multi-collinearity a concern with SVM and Random Forest when using binary/categorical data?

With SVMs? Generally no. A distance-based kernel, e.g. the RBF kernel, definitely does not have this issue. Linear or polynomial kernels or so on do use linear combinations in that way, and so if you don't regularize, you can run into the same multicollinearity problems as you do with OLS. But you should almost never not regularize an SVM; the standard $L_2$ regularization is equivalent to doing ridge regression, which handles most of the issues with multicollinearity, as noted in this thread linked to by @DJohnson above.
• Logistic regression is similar to SVMs here in that it's usually going to be $L_2$-regularized. If you only care about predictions, it's fine; if you care about inference for the coefficients, it can be a problem, in the same way it's a problem for ridge regression. – Dougal Nov 28 '15 at 22:18
• @prashanth That other post is talking only about linear regression-type models, where rank-deficiency is an issue. For random forests, it's not an issue, and sometimes advantageous to include all k. For SVMs with distance-based kernels, e.g. Gaussian RBF, it's preferable to include all of them because it makes the distances between different categories consistent; otherwise your "default" category is distance $1$ to everything, but all the others are $\sqrt 2$ apart. For linear-kernel SVMs, it can possibly be a problem to have all k, but probably not. – Dougal Feb 25 '16 at 6:47