3
$\begingroup$

I am aware that a feature with too many levels might be bad for a number of algorithms (e.g. Logistic Regression).

A typical approach to fix this would be to group the categories with a frequency lower than a predefined threshold in a single category (e.g. "Other").

But, technically speaking, why is having too many levels on a feature exactly bad? Are there any algorithms for which this fix isn't needed, i.e. that address this issue internally?

Improve this question
$\endgroup$
  • $\begingroup$ Your 1st paragraph & the 1st sentence of your 3rd paragraph seem inconsistent with each other. Moreover, I don't know that logistic regression is in any way special WRT this issue. $\endgroup$ – gung - Reinstate Monica Dec 5 '18 at 18:12
  • 1
    $\begingroup$ As a general matter, it's hard to estimate effect sizes when you don't have much data. Suppose you're randomly sampling 5000 people from all of the USA. Many of America's ~3000 counties will have 0 or 1 people in your sample, purely due to the vagaries of random sampling. It will be challenging to say anything definitive about whatever phenomenon you're studying at the county level. But aggregating to the level of the 50 states might informative, and more precise, because you'll have more data about each state (but you won't easily make county-level inferences). $\endgroup$ – Sycorax♦ Dec 5 '18 at 18:15
  • $\begingroup$ Aggregating to "other" category is not the only (or best) solution. In neural networks you can use entity embeddings. General solution that works well is to use hashing trick. $\endgroup$ – Tim♦ Dec 5 '18 at 18:31
  • $\begingroup$ @gung Why do you think so? In the 1st paragraph im stating that too many levels might be bad, while on the 3rd one I'm asking what are the reasons for being bad. $\endgroup$ – GRoutar Dec 5 '18 at 19:05
  • 2
    $\begingroup$ Well, each new feature level increases the number of parameters with 1, and too many parameters is difficult. You can find interesting ideas in stats.stackexchange.com/questions/146907/… $\endgroup$ – kjetil b halvorsen♦ Dec 5 '18 at 20:38
2
$\begingroup$

Each new feature level increases the number of parameters by 1, and too many parameters makes estimation difficult. Sometimes it is said that in linear regression, you should have ten observations for each parameter. See how many parameters can be estimated and more details in the book there cited.

You propose to group the categories with a frequency lower than a predefined threshold in a single category (e.g. "Other"). That can be a bad approach, just because two levels have few observations doesn't mean they have the same effect on the response. Some other method is called for, and there are good suggestions here: Principled way of collapsing categorical variables with many levels?. Also search for the tag.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.