With the goal of selecting predictors for a 4 level outcome variable I want to apply LASSO for predictor selection. Some continuous variables are related to each-other and should all be in the final model, or none.

Context: The dataset consists of around 1700 rows of data ($n$) with around 30 predictors (a mix of continuous predictors, binary categorical and multilevel categorical ones). As stated, the outcome/dependent variable is a 4-level categorical one (not ordered). The frequency of the 4 outcome groups is around 600/250/250/600.

Now I recon some associations between continuous predictors and the outcome are not linear. But, I do not have an idea of the functional form of these possible non-linear associations. So, I wanted to use (restricted cubic) splines to allow for non-linear associations for the continuous variables.

The problem is that when creating a spline this is actually a data transformation where a continuous variable $x$ is used to create 'spline-variables' based on splineknots. Simply put, when the knots are set at 30 and 60, the set of spline variables would consist of $x$, $x'$ and $x''$, where:

  • $x$ is the original continuous variable;
  • $x'$ is $0$ when $x<30$, and a function $f(x-30)$ when $x>=30$
  • $x''$ is $0$ when $x<60$, and a function $f(x-60)$ when $x>=60$

My strategy now is to obtain restricted cubic spline data through a regular multinomial regression, extract the model-matrix, and feed this into a LASSO function. As stated I'd want to keep the sets of splinevariables together, or dropped completely (just like the dummies for a categorical variable would). However, I have not been able to find a proper function (in R) which does grouped LASSO for multinomial regression. Moreover, the elastic net functions I've found do not support spline fitting within the function, do not support grouping of predictor variables, and/or do not support multinomial regression models.

For example, the 'glmnet' documentation does not mention grouping of variables and the 'gglasso' and 'grplasso' are for binary outcomes only (or so they seem);

in short: is it possible to perform grouped LASSO for a multinomial regression model? Or fit splines within these functions? And if so, how? (which software - I'd prefer an R-package - allows this?)

Ps. This question is related to this one, but looking at the comments and answers there I am definitely looking for something else. The 'type.multinomial = "grouped"' option in glmnet does not keep specific variables together, but instead keeps a single variable in for all outcomes of a multinomial regression, if it is retained for any one of the outcomes (i.e. even when using this option I still see certain splinevariables dropped while related splinevariables are retained). Further, as stated, the answer provided there (using gglasso) does not apply to multinomial regression.

  • $\begingroup$ If you know what objective you have in mind, why do you need a package to do it? Just use an optimization library. If you want to use R, you can use the package here (web.stanford.edu/~boyd/papers/pdf/cvxr_paper.pdf) for optimization, for instance. $\endgroup$
    – user795305
    Dec 6 '17 at 16:13
  • $\begingroup$ @user795305 could you elaborate with an answer how to? I'm not that proficient with writing my own optimization functions $\endgroup$
    – IWS
    Dec 6 '17 at 16:14
  • $\begingroup$ @user795305 I will try though! Thanks for the link $\endgroup$
    – IWS
    Dec 6 '17 at 16:21
  • 2
    $\begingroup$ In one recent project of mine, whose purpose was solely for prediction, I counted and reported on the terms with nonzero coefficients. One way to interpret this is that selecting any subset of the spline terms for a variable is tantamount to selecting a particular family of nonlinear transformations of that variable. $\endgroup$
    – whuber
    Dec 6 '17 at 16:33
  • 2
    $\begingroup$ If you use even one level of a categorical variable you are keeping that variable in the model. It's analogous to the spline: if you keep even a single spline term in the model, you are using the associated variable in the model. It's simply a matter of how you will express it. In effect, the Lasso is trying to tell you how to bin and encode the levels of your categorical predictors. For instance: when using effects coding, the zero coefficients indicate those levels to combine with the reference level. (This interpretation potentially could offer some insight into the Lasso results.) $\endgroup$
    – whuber
    Dec 7 '17 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.