I use lasso logistic regression and want to test the influence of a suqared (X1^2) variable on the outcome.

Assume, I put all variables in my model and it looks sth. like this:

Y (0/1)

  1. X1
  2. X1^2
  3. X2
  4. X3

When I now perform a lasso regression, variables X2 and X3 remain in the model. However, variable X1 gets kicked out (zero) and X1^2 remains in the model. Assume it looks sth. like this.

  1. 0
  2. 0.34
  3. 0.67
  4. 0.12

Question 1: Did I implement the squared variable correctly (X1 and X1^2)?

Question 2: Does it make sense that X1 is out of the model bit X1^2 is in the model. Can one interpret this?

Question 3: Should I fix X1 at a certain level, or protect it from shrinkage?


I am of two minds with this. In a traditional modeling approach, one would never include a product feature without including their lower level features, and in the case of a quadratic feature that means you'd include the linear feature as well. This preserves the interpretation of the coefficients, which is valuable for inference.

LASSO differs from traditional regression in two ways: firstly, it is focused on prediction and secondly (more importantly), a zero value for a coefficient does not reflect a belief that there is no association with the outcome of interest; instead, it merely says that the effect is so small, it is effectively zero. In both cases, we have a rationale for having zero valued coefficients for lower level features.

On the other hand, I would be concerned that we have introduced some issues with internal validity. As we know in time series analysis, higher level polynomial terms have a tendency to "balloon" outward when a tendency toward asymptotes is more of a believable mechanism, like in biological homeostasis, growth curves, population dynamics, economics, etc. etc. I don't believe that including lower level features will do magic in this regard, but I have a strong belief it will provide better external validity (i.e. in datasets whose structure may differ markedly from the training and validation datasets).

So to answer the direct questions:

1) I can't tell, there is no code

2) Yes and no. No, there is no believable interpretation of the quadratic term without the lower level linear term. To describe why LASSO obtained this, you can say that the gradient of the quadratic surface in this variable was very close to zero at the origin.

3) I would recommend the following two-step approach: identify significant features using LASSO and CV obtained tuning parameter. Then, refit the model using the corresponding regression model including all the significant terms you identified in the original LASSO as well as the lower level terms.

  • $\begingroup$ perfect, thank you very much. This is all a lot more complicated than I thought it would be. At point 3), what exactly do you mean by corresponding regression model? $\endgroup$ – Toby_Shoby Jun 6 '16 at 16:04
  • 1
    $\begingroup$ @user3519420 Suppose you are fitting a linear regression model for an outcome variable on, say, 1000 features. The resulting cross validation identifies features X1, X10, and X25 as significantly non-zero in the penalized regression. However, suppose X10 is the quadratic term you created from some variable, say "age". Then you'd fit a "final" model from the features X1, X25 (whatever those are), and add age as well as age squared. $\endgroup$ – AdamO Jun 6 '16 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.