Handling categorical predictors in logistic regression, linear regression and SVM

Question

I want to know how I can handle categorical variables in logistic regression, linear regression and SVM. The categorical variable has four categories 1,2,3 and 4. However, it doesn't mean 4 is like 4 times 1. It's just a name. Why people say that for SVM I need to use four dummy binary variables encoding the category, but for linear and logistic regression, I do not need to do anything? Why is that any insights.

Answer 1

You should be using dummies for all of these approaches for exactly the reason you specified: the numeric value by which the categories happen to be encoded doesn't capture similarity (e.g. 4 isn't closer to 3 than it is to 1).

If you have a model with linear components of the form $\beta_i x_i$ (all three of the methods you mentioned do, even kernel SVM), then by not using dummy variables you are stating by design that 4 is closer to 3 than it is to 1. Suppose you have 3 instances with values 1, 3 and 4, then their contribution to (parts of) the prediction is $$\begin{align} x_i = 1 \rightarrow\ &\beta_i, \\ x_i = 3 \rightarrow\ &3\beta_i, \\ x_i = 4 \rightarrow\ &4\beta_i, \end{align}$$ and hence the contribution of $x_i=4$ appears more similar to $x_i=3$ than to $x_i=1$, even though they aren't. With dummies you completely avoid this issue.

Techniques like decision trees don't require such encoding, because they generate splits rather than use the numeric values directly (the splits will land somewhere between the categories and thus effectively separate them).