1
$\begingroup$

In order to overcome over-fitting during a regression process over categorical features, one can either

1) Apply L1/L2/Elastic regularization during the regression, for example as answered here

When to use regularization methods for regression?

2) Filter out categories which appear less than X times, where X is at least 10-20 ,for example

Minimum number of observations for multiple linear regression

The advantage in method 2 is that I can set with this rule of thumb approach the value of X to be 10-20, while the regularization method requires a costly computational cross validation step to be able to choose properly the correct regularization parameter, which would in fact multiply the regression running time by a factor. In addition, it greatly reduces the number of features when the categories have a long tail distribution - This can mean a big decrease in running time and memory consumption.

What is the common practice? When should I prefer one method over the other? Is the second method an acceptable one, or is the correct way to always perform some sort of regularization?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.