In order to overcome over-fitting during a regression process over categorical features, one can either

1) Apply L1/L2/Elastic regularization during the regression, for example as answered here

When to use regularization methods for regression?

2) Filter out categories which appear less than X times, where X is at least 10-20 ,for example

Minimum number of observations for multiple linear regression

The advantage in method 2 is that I can set with this rule of thumb approach the value of X to be 10-20, while the regularization method requires a costly computational cross validation step to be able to choose properly the correct regularization parameter, which would in fact multiply the regression running time by a factor. In addition, it greatly reduces the number of features when the categories have a long tail distribution - This can mean a big decrease in running time and memory consumption.

What is the common practice? When should I prefer one method over the other? Is the second method an acceptable one, or is the correct way to always perform some sort of regularization?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.