0
$\begingroup$

I have a dataset containing a fair amount of continuous and categorical variables. I one-hot encode these variables to be used in various machine learning algorithms.

Let's presume a variable has n categories, which we one-hot encode into n columns. If we work with penalized models, we want to standardize all variables. However, when we standardize a one-hot encoded variable, for one variable, we get n standardized columns. Does this mean we are giving an advantage to categorical variables in terms of regularization, especially if a variable has many categories?

This problem seems especially relevant when using KNN algorithms (not only for prediction but also imputation). Without standardization, the distances would be biased towards high-valued variables. However, the distances seem to become biased towards categorical data when we standardize, especially if the variable has many categories. If, say, we have a binary categorical variable with an equal number of samples in each category, after standardization 0's would be replaced with -1's, and 1's would remain 1's. Then, the Euclidean distance between two samples with a different categorical value ([-1, 1] and [1, -1]) would be 2*sqrt(2), whereas the distance between a standardized continuous variable would likely be closer to one or two standard deviations (1-2), presuming it's normally distributed (is that an important assumption in this case?).

Following the above logic, a simple solution that comes to mind is dividing each one-hot encoded column by the number of categories. So, in the above example, the distance between two samples with a different binary category would be sqrt(2) ([-0.5, 0.5] and [0.5, -0.5]). That way, the total distance between samples seems to be more evenly distributed between variables, with less bias towards categorical variables with a large number of categories. Another solution that comes to mind is, instead of standardizing categorical variables, simply replacing the 0's with -1/n, and the 1's with 1/n.

Naturally, simply treating all of these ideas as hyperparameters would likely be the "best" solution when trying to get the best model, but I'm interested if there's any literature on the subject. Has anyone had any experience with this?

Thanks.

Improve this question
$\endgroup$
2
  • $\begingroup$ Why not do dimensionality reduction first? For continuous features, PCA. For categorical (polytomous) features, Multiple Correspondence analysis (MCA) or Categorical PCA. Also if you have many categories in a feature and some categories are tiny percent, first merge such ones into "other" category. $\endgroup$
    – ttnphns
    Commented Apr 13, 2023 at 16:45
  • $\begingroup$ @ttnphns While that is a possibility, it seems to me that such an approach potentially needlessly reduces the amount of available information and, more importantly, makes interpretation less feasible. $\endgroup$
    – bob_cart
    Commented Apr 14, 2023 at 7:23

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.