I have a not-so-large data set (n = 3000, p = 8), where my primary explanatory variables of interest are categorical with, in a few cases, many (>10) levels. Apologize for the lack of specificity but I cannot disclose much on the application - my main question is about a methodology which I am trying to employ.
A response to a question for a similar problem (Principled way of collapsing categorical variables with many categories) led me to Gertheiss, J. and Tutz, G. "Sparse Modeling of Categorial Explanatory Variables", found here: http://projecteuclid.org/download/pdfview_1/euclid.aoas/1294167814
This methodology seems to be exactly what I am looking for. The paper proposes a modification of the penalty term for the LASSO and operates on differences between dummy variables to both cluster levels within a given variable and also perform variable selection. Further, the results seem easy to interpret and the visual showing the clustering path of levels resonates well with the less technical.
Has anyone employed this, or a related methodology successfully? I am trying to conduct a transformation of variables as proposed in the paper and then simply run the lasso via lars in R. Here are the problems I am running into:
- What is the specification of the X matrix? Typically, we would expect (a - 1) dummy variables for predictor A with a levels. In this case, I believe the model has an overall intercept term and also over-specifies the explanatory variables (i.e., each level has a dummy variable).
- How is the transformation matrix, U (pg 2157), defined for nominal predictors? The definition makes sense for ordinal categories to me. That seems clear. The definition on the surface makes sense for nominal categories to me. However, in practice, I get caught up in the details. The definition I thought made sense ended up in a non-square matrix, with caused problems calculating the inverse. I am under the impression that this is a p x p matrix, where p = number of parameters (including intercept and all dummy variables). This is (what I believe) is my main problem.
If anyone can shed some light into applying this methodology through R, that would be much appreciated. I am close, but still fighting through the details.