1
$\begingroup$

I'm running a multinomial logit to predict the outcome of a categoric response variable. I have both continuous and categoric independent variables, and I know it's bad practicde to bin the continunous ones. For the categoric, however, I've seen it's very used (and it makes sense, specially since I have a lot of observations) to make dummies out of the n-1 categories. I also don't have too many categories in each categoric variable (less than 20), and not many categoric variables in total.

But I was wondering which is better: to simply make dummies for the n-1 categories, or to create bins using any binning method, like IV, WoE, Chi square, KS?

Intuitively I feel like creating dummies would be better since you're capturing precisely the effect of each category, while with the bins you're always losing a bit of predictive power because you're joining them.

$\endgroup$
6
  • $\begingroup$ I wouldn't state that binning a continuous variable is always a bad practice. It depends. $\endgroup$
    – ttnphns
    Sep 7 '20 at 19:52
  • $\begingroup$ Yeah, "always" is never true haha but most of the times. The thing is that for my study I decided that it is not suitable. $\endgroup$
    – amestrian
    Sep 7 '20 at 19:55
  • $\begingroup$ By common definition, binning aka discretization aka categorization is for continuous variables only. What you are asking about categorical variables is combining them into bigger, fewer categories. This should go with tag [many-categories], I might suggest. $\endgroup$
    – ttnphns
    Sep 7 '20 at 19:57
  • $\begingroup$ ah, thank you! I have always called it the same for both types of variables. Just changed the tags. $\endgroup$
    – amestrian
    Sep 7 '20 at 20:02
  • 1
    $\begingroup$ See stats.stackexchange.com/questions/146907/… $\endgroup$ Sep 7 '20 at 20:31
1
$\begingroup$

Note there's already a number of good questions on this topic, like this one or this one.

In those, some alternative options to one-hot-encoding (aka dummy variables) are mentioned. These include to allow for partial pooling (i.e. especially rare classes would be pulled towards being an average category, while large classes that do have a lot of data showing different behavior than other classes end up being "alllowed to be different") using random effects.

Another option that's also mentioned in one of the answers is target encoding. This is also very popular on Kaggle. To adapt this for multinomial logistic regression, you could in the regression equation for each target category have a covariate that target encodes that target category. I.e. you create a covariate that captures for what proportion of cases this class level has resulted in this target category, but you shrink these proportions towards the overall proportion (and possibly even the proportion for each target category towards an equal proportion for all target categories). The extent of shrinkage is a tunable hyperparameter.

If there's some sensible way of doing it, you could also consider finding a suitable embedding for the features. E.g. one can create these using some pre-training task (e.g. like one creates Word2Vec/GloVe etc. word embeddings), or one could project properties of the classes (assuming you know them) into a low dimensional space using some technique like UMAP or t-SNE. However, both of those options require some extra information and might not always be possible. If you train a neural network for tabular data, you could of course create (or fine-tune) these as you train on your target task - or even train on that task and then use these embeddings in another model.

$\endgroup$
0
$\begingroup$

As @ttnphns indicates in the comments, you are asking whether you should collapse categories of a categorical variable into fewer "levels" of the category. There is nothing wrong with this approach and is sometimes a necessity. For example if you have a categorical variable with, say, 1000 categories, but you can logically collapse these into a only two categories that makes sense in the context of your analysis, then you should do so. Indeed, using the original 1000 categories, generally uses $p-1=999$ degrees of freedom in your model. However, if you then collapse these categories into only 2 categories, this will consume only a single degree of freedom: you model will then only contain 1 dummy variable in your model, rather than 999 dummy variables.

$\endgroup$
2
  • $\begingroup$ I forgot to write that there aren't many categories (less than 20) in any of my categoric variables. And I have a lot of observations (17000+) so I don't think the degrees of freedom in this case would be a problem. Do you have any suggestion for the method to choose for collapsing the categories? $\endgroup$
    – amestrian
    Sep 7 '20 at 20:18
  • $\begingroup$ I doesn't really matter if you have a lot of categories. If the intent of your model is for it to be descriptive, rather than predictive, I'd argue for collapsing categories if it makes sense from a contextual perspective. If your intent is purely predictive, then I'd chose the method that results in the smaller mean squared predicted error. $\endgroup$ Sep 7 '20 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.