LASSO with interaction terms - is it okay if main effects are shrunk to zero?

Question

LASSO regression shrinks coefficients towards zero, thus providing effectively model selection. I believe that in my data there are meaningful interactions between nominal and continuous covariates. Not necessarily, however, are the 'main effects' of the true model meaningful (non-zero). Of course I do not know this since the true model is unknown. My objectives are to find the true model and predict the outcome as closely as possible.

I have learned that the classical approach to model building would always include a main effect before an interaction is included. Thus there cannot be a model without a main effect of two covariates $X$ and $Z$ if there is an interaction of the covariates $X*Z$ in the same model. The step function in R consequently carefully selects model terms (e.g. based on backward or forward AIC) abiding to this rule.

LASSO seems to work differently. Since all parameters are penalized it may without doubt happen that a main effect is shrunk to zero whereas the interaction of the best (e.g. cross-validated) model is non-zero. This I find in particular for my data when using R's glmnet package.

I received criticism based on the first rule quoted above, i.e. my final cross-validated Lasso model does not include the corresponding main effect terms of some non-zero interaction. However this rule seems somewhat strange in this context. What it comes down to is the question whether the parameter in the true model is zero. Let's assume it is but the interaction is non-zero, then LASSO will identify this perhaps, thus finding the correct model. In fact it seems predictions from this model will be more precise because the model does not contain the true-zero main effect, which is effectively a noise variable.

May I refute the criticism based on this ground or should I take pre-cautions somehow that LASSO does include the main effect before the interaction term?

Answer 1

One difficulty in answering this question is that it's hard to reconcile LASSO with the idea of a "true" model in most real-world applications, which typically have non-negligible correlations among predictor variables. In that case, as with any variable selection technique, the particular predictors returned with non-zero coefficients by LASSO will depend on the vagaries of sampling from the underlying population. You can check this by performing LASSO on multiple bootstrap samples from the same data set and comparing the sets of predictor variables that are returned.

Furthermore, as @AndrewM noted in a comment, the bias of estimates provided by LASSO means that you will not be predicting outcomes "as closely as possible." Rather, you are predicting outcomes that are based on a particular choice of the unavoidable bias-variance tradeoff.

So given those difficulties, I would hope that you would want to know for yourself, not just to satisfy a critic, the magnitudes of main effects of the variables that contribute to the interaction. There is a package available in R, glinternet, that seems to do precisely what you need (although I have no experience with it):

Group-Lasso INTERaction-NET. Fits linear pairwise-interaction models that satisfy strong hierarchy: if an interaction coefficient is estimated to be nonzero, then its two associated main effects also have nonzero estimated coefficients. Accommodates categorical variables (factors) with arbitrary numbers of levels, continuous variables, and combinations thereof.

Alternatively, if you do not have too many predictors, you might consider ridge regression instead, which will return coefficients for all variables that may be much less dependent on the vagaries of your particular data sample.

Answer 2

I am late for a party, but here are few of my thoughts about your problem.

1. lasso selects what is informative. Lets consider lasso as a method to get the highest predictive performance with the smallest number of features. It is totally fine that in some cases, lasso selects interaction and not main effects. It just mean that main effects are not informative, but interactions are.

2. You are just reporting, what you found out. You used some method and it produced some results. You report it in a transparent manner that allows reproducibility. In my opinion, your job is done. Results are objective, you found what you found and it's not your job to justify, why you didn't find something else.

3. All units are arbitrary. Interactions are just units. Lets say you study colors. Colors can be included in your model as a wave length, or a log wave length, or as 3 RGB variables, or as an interaction of a hue and tint and so on. There is no inherently correct or incorrect representation of colors. You will choose the one that makes most sense for your problem. Interactions are also just units that you can use arbitrarily. Area of a window, is just interaction of its height and width, should you include height and width of a window in your model? Velocity is just interaction of mass and speed. And Speed is just interaction of time and distance. Manhours is just interaction of time and number of people working. Mathematically treatment dose * age is the same as height * width. The "you have to always include main effects" saying is overrated.

4. lasso does not approximate real model, it's not meant for inference and selected variables are unstable. If you have correlated informative predictors, lasso tends to choose one and push the others to 0, therefore your model will omit significant proportion of informative variables. Also, as was pointed out in the comments, if you find the best lambda in crossvalidation, lasso will choose more variables than a real model has. Another issue is, that selections from lasso are unstable. So if you run lasso again on a different sample from a population, you will end with a different set of selected variables. Hence don't put much weight on which variables are selected. Also, the betas are biased, and therefore cannot be used for a classical parametric hypothesis testing. However, there are ways around it (next point)

5. inference with lasso. Lasso can be use to make a inference on predictors. Simplest way is to bootstrap it and count how many times each variable is selected, divide by number of resamples, and you have your p-values. P in that case is a probability of a variable being selected by lasso. You can still end up with significant interaction effects and insignificant main effects, but that's not a problem, it can happen with normal hypothesis testing as well. Great treatment of this topic is in the Hastie et. al. free book: Statistical Learning With Sparsity, chapter 6 http://web.stanford.edu/~hastie/StatLearnSparsity/ The bootstrap can be performed for whole range of lambda values which will result in a stability path for all variables. This can be extended with a stability selection approach to find a set of significant variables corrected for family wise error. http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2010.00740.x/abstract There are also some other methods for inference with lasso, that might be useful. Namely adaptive lasso or desparsified lasso. Review with R implementation is here DOI: 10.1214/15-STS527 or IMO more accessible explanation in the Buhlmanm, van de Geer Book: Statistics for High-Dimensional Data http://www.springer.com/la/book/9783642201912

6. Other lasso related things to be aware. As far as I know ridge or elastic net tends to outperform lasso. If there is a domain knowledge about variables, group lasso or sparse group lasso can be used in order to force lasso to either keep or discard the whole group of predictors instead of treating them individually (e.g. gene paths, dummy coded factor variable). For spatial or ordered data fused lasso can be used. Randomized lasso, introduced in the stability selection paper mentioned above, tends to produce sparser models with the same performance as a standard lasso.

Answer 3

For the Lasso, if your using it in a predictatory setting then the only thing that matters is how well the cross-validated results are. If you are trying to conduct inference around your effects then read the paper about "Double Lasso".

Answer 4

I have an application where I specifically want small number of main effect to be not penalized. Let Y = X.mainbeta + X.interbeta.inter + eps

a) fit.Y = OLS(X.main,Y). Let tilde.Y = Y - predict(fit.Y,X.main) b) fit[,j] = OLS(X.main, X.inter[,j]) for j = 1...k. Let tilde.X.inter[,j] = X.inter[,j] - predict(fit.j,X.main) c) fit = Lasso (tilde.X.inter,tilde.y) . The coefficient on main effect equals fit.Y - coef(fit)*fit[,1:dim(X.inter)[2]]. The coefficient on interaction effect equals coef(fit)

In steps a and b, no need to do sample splitting. That works for me!