First of all, this question is related to this one: Principled way of collapsing categorical variables with many levels? but I think the scope of the answers I'm looking for is different.
Just to present a problem, assume we want to perform a logistic regression in which we have two categorical variables as input, each with 3 categories. We code them as dummies, use one category as reference and are left with 4 dummy variables X1 and X2, Y1 and Y2.
The results of the logistic regression could look like this:
Now, we know that all the dummy predictors are statistically significant and different to to the reference category but what about between them? It may seem reasonable to merge categories X1 and X2 but leave Y1 and Y2 as is.
The question I linked in the beginning touches on this issue and, based on the many answers, it would seem that the best approach is to adopt a custom-made regularization penalty that would penalize -along with coefficient sizes- differences between coefficients for dummies resulting from the same categorical variable (and therefore somewhat different from the implementations of fused lasso I've seen).
As far as I can tell no implementations of something similar exist (and if I'm wrong please correct me). My question is this: How was this dealt with before Lasso or how is this dealt today when such algorithms aren't always readily available? Surely this is a problem/question that comes up often?
Is there a rule of thumb that would provide some threshold or direction on when we can merge categories and when we shouldn't?