0
$\begingroup$

First of all, this question is related to this one: Principled way of collapsing categorical variables with many levels? but I think the scope of the answers I'm looking for is different.

Just to present a problem, assume we want to perform a logistic regression in which we have two categorical variables as input, each with 3 categories. We code them as dummies, use one category as reference and are left with 4 dummy variables X1 and X2, Y1 and Y2.

The results of the logistic regression could look like this:

Predictor Coef St. Error
X1 0.5 0.2
X2 0.6 0.25
Y1 0.8 0.05
Y2 1.2 0.07

Now, we know that all the dummy predictors are statistically significant and different to to the reference category but what about between them? It may seem reasonable to merge categories X1 and X2 but leave Y1 and Y2 as is.

The question I linked in the beginning touches on this issue and, based on the many answers, it would seem that the best approach is to adopt a custom-made regularization penalty that would penalize -along with coefficient sizes- differences between coefficients for dummies resulting from the same categorical variable (and therefore somewhat different from the implementations of fused lasso I've seen).

As far as I can tell no implementations of something similar exist (and if I'm wrong please correct me). My question is this: How was this dealt with before Lasso or how is this dealt today when such algorithms aren't always readily available? Surely this is a problem/question that comes up often?

Is there a rule of thumb that would provide some threshold or direction on when we can merge categories and when we shouldn't?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Could you explain what substantive problem you would be trying to solve by merging categories? Although you write this may "seem reasonable," what is the basis for that and for which purpose? For some objectives such steps might be useful but for others (especially if you plan on doing formal testing or seek explanatory power) they could be biased and ad hoc. $\endgroup$
    – whuber ♦
    Feb 7 at 20:29
  • 1
    $\begingroup$ I guess the main idea would be to be more parsimonious and avoid overfitting in cases when you cannot be confident -statistically speaking- that observations falling in one category, e.g. here X2, are expected to behave differently in the future compared to those in some other, e.g. X1, category. I understand that this could be biased and ad hoc, this is why I'm looking to see if something akin to a 'standard practice' on this matter exists. $\endgroup$
    – danton
    Feb 7 at 20:44

1 Answer 1

Reset to default
1
$\begingroup$

Quantitative sociology/psychology is probably a better guide than the statistics literature on this question. For example, this article from a journal on structural equation modeling SEM: Collapsing Categories is a good introduction to collapsing categories if you are interested in understanding the covariance structure between the variables (i.e. if you are more interested in understanding how to columns in an X matrix relate to each other).

Improve this answer
$\endgroup$
1
  • $\begingroup$ The idea of a custom-made penalty function is the best approach by far. Avoid any ad hoc solutions other than pragmatic ones that look at low cell frequencies and do not rely on any analysis that uses Y, which is double dipping. $\endgroup$
    – Frank Harrell
    Apr 12 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.