3
$\begingroup$

I ran a elastic net regression path for a multinomial model using the glmnet package in R. My response variable has 3 levels (0, 1 and 2 for 3 different stages of a disease).

Here are the coefficient output from the elastic net

# Stage0
# (Intercept) -0.14301343
# Age          -0.16305154
# Q1       -0.04925107
# Q2       0.01030530
# Q5      0.11559569
# Q9      0.01927071

#Stage1
# (Intercept) -0.9571839
# Q5           0.4063393

#Stage2
# (Intercept)  1.10019732
# Q2      -0.04281563
# Q4      -0.07533350
# Q6      -0.01961934
# Q7      -0.15523562
# Q9      -0.03297465
# Q17     -0.06386999

My predictor variables are age, and Q1-Q18 for 18 items on a psychiatric questionnaire. I see that the elastic net shrunk certain coefficients to 0. My questions are

1) In the output above, the coefficients are separated by my response level (stage0, stage1, or stage2). What exactly does this mean? Is it that the coefficients under Stage0 are for the model when the multinomial model uses Stage0 as the reference level? How do I interpret these coefficients?

2) Can I take these remaining variables and fit a multinomial regression model? So instead of using all of my original 19 predictor variables, I can just focus on those that were selected by elastic net and do model building from there?

$\endgroup$
1
  • $\begingroup$ 1) I recommend asking the first question as its own question, as interpreting the coefficients of a multinomial logistic regression is different from how to deal with coefficients set to zero by LASSO. // 2) Your $y$ sounds ordinal. In such a case, ordinal models might make more sense than multinomial. $\endgroup$
    – Dave
    Commented May 6, 2022 at 18:41

1 Answer 1

1
$\begingroup$

(I will address the second question.)

NO

While you no longer have to collect data on the variables who coefficients are estimated as zero, eliminating those variables and refitting will (or at least can) change the other coefficient estimates. During the minimization of the LASSO loss function, the features that wind up "dead" still contribute by setting their coefficients to be zero.

I find it plausible that the math could work out to give that the penalized coefficient estimates happen to be the same as the unpenalized coefficient estimates for only the surviving features. That would be an elegant result, but it happens not to be correct.

$\endgroup$
5
  • $\begingroup$ (+1) for what it's worth, the point isn't that the "dead" coefficients still contribute. Elastic net both shrinks and selects, by design. The answer to whether the coefficients of a relaxed elastic net are the same as the coefs from the elastic net is "no" because of the shrinkage. $\endgroup$
    – Ben
    Commented May 6, 2022 at 18:50
  • $\begingroup$ @Ben I do not follow your argument. Could you please elaborate? $\endgroup$
    – Dave
    Commented May 6, 2022 at 18:51
  • $\begingroup$ I think it's hard to properly elaborate in the comments. If you post a question on why "elastic net" or "elastic net followed by unpenalized estimation on the estimated support" are not equal, I'll post a thorough reply. $\endgroup$
    – Ben
    Commented May 6, 2022 at 19:04
  • $\begingroup$ @Ben It seems that I've even responded to such a question! Perhaps you can post an answer that is more thorough (though I like my comment about “You will be zero no matter what"). $\endgroup$
    – Dave
    Commented May 6, 2022 at 19:14
  • $\begingroup$ Thanks for the link. I posted a reply. I hope it's clear--there is some mathematical notation, but there's no math actually being used. $\endgroup$
    – Ben
    Commented May 6, 2022 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.