New answers tagged

0

Depending on the data and the model and how performance is assessed, new data with a new feature could either improve or worsen the performance.


0

You say you represented the category levels by integers. That is not the same as using dummy encoding! and should not be done. This error probably explains why a decision tree gave much better results. Try again, but now using dummy encoding. You hint in comments about 75+ categories (better to say levels), why is that a problem? If you represent the ...


1

There is in general no reason to believe that just because some levels are infrequent, they have the same effect on the outcome variable. So I would be doubtful of your proposal. Maybe go for some regularization approach, for your problem I would try the fused lasso. Some useful discussion in Principled way of collapsing categorical variables with many ...


1

You could use the categorical variable with its 500 levels as is, but then use regularized logistic regression. In Principled way of collapsing categorical variables with many levels? one idea is to used the fused lasso, but there are other possibilities. I cannot see how the power-law distribution is relevant, and your idea of merging all but the 10 most ...


1

Assuming that by binary encoding you mean the one explained here, I would advice against using it. Seems an ill-advised idea, I will explain why. First explaining shortly the idea: Suppose (only for simplicity) your categorical variable have $p=2^q$ levels, for the example I take $q=3$. Then code the levels with the binary numbers $0=000_2, 1=001_2, 2=...


Top 50 recent answers are included