Does Categorical Variable need normalization/standardization?

Question

since we do normalize as 10kg >>> 10 grams or 1000 >> 10. so incase of one hot encoding eg male=0 and female =1, are we giving more weight to female as 1>0 for training our models?

Answer 1

The 0/1 encoding of male/female doesn't by itself put more weight on females versus males; it's not really different from having a value of 0 versus 1 (or 1 versus 2) in a continuous predictor. It's just a difference of 1 unit in the predictor value. As @Tim rightly points out, for many machine learning approaches normalization is not required and there is no issue about weighting.

There is, however, a potential problem when your modeling method requires all predictors to be on the same scale. Examples are principal-components analysis (PCA) and penalized linear approaches like LASSO, ridge, or their hybrid elastic net. The default, in some implementations at least, is to normalize all predictors including the categorical predictors. That's where you can have difficulties.

If you don't normalize the categorical predictors, are they on the same scales as the continuous predictors, as your penalization method implicitly assumes? If you do normalize 2-level predictors like male/female, the normalized values can depend heavily on the class frequencies in your sample. So normalization of predictors having class imbalance can alter weighting. For multi-level categorical predictors the issue is even more vexing as the choice of the reference level can affect the normalized values.

This page goes into more detail. There is no one-size-fits-all solution to this issue in PCA and penalized regression; intelligent application of your subject-matter knowledge might be best.

Answer 2

Machine learning algorithm is a function of the inputs, that predicts the outputs. There are many different algorithms. You seem to assume linear model, where $y = X\beta + \varepsilon$, so the result of $y$ would linearly depend on $X$. Notice however, that even with such model if $\beta$ is negative, then decreasing $X$ would lead to increasing $y$.

Moreover, most of the machine learning models learn non-linear functions, so there is no such linear dependence. For example, decision tree is a series of if ... else ... statements based on learned thresholds if X > c then ... else ..., so it doesn't matter for the model what are the actual values as long as it can pack them into meaningful "buckets" of similar values. Neural networks achieve non-linearity by using redundant weights and stacking multiple layers. If you use $k$ nearest neighbors, it only looks at similarities between your samples, so bigger/smaller relation does not affect it in this case.

Finally, normalization/standardization does not affect ordering of values. So if $x_1$ is larger then $x_2$, after normalization or standardization they both would have potentially different values, but the relation between them would not change.

Answer 3

Normalization/standardization of features is done to bring all features to a similar scale. When you one hot encode categorical variables they are either 0/1 hence there is not much scale difference like 10~1000 hence there is no need to apply techniques for normalization/standardization.