When in Adaptive LASSO process does it make sense to constrain control variable lambdas to 0?

Question

Lets take this example for how to conduct an adaptive LASSO. Essentially an initial model is fit using ridge regression. Then, a LASSO is fit, in which the value of lambda is tuned individually for each predictor based on its coefficient in the ridge regression.

Suppose I would like to include a control variable.

I know I can alter the model such that lambda doesn't get applied to a certain variable by setting the penalty factor to be 0 for those variables (i.e., the candidate control variables).

My question is: Does it make more sense to do this in both the initial ridge model and the rescale model (i.e., ride1_cv and alasso1_cv in the example). Or just in the rescale model?

---

Code for setting lambdas to 0:

For example, in ridge1_cv I would:

#create a vector of 0 for my control variable and 1s everywhere else, and add a penalty factor term to the model:

penf <- c(0, rep(1, times = (ncol(d)-1))

ridge1_cv <- cv.glmnet(x = x_cont, y = y_cont,
                       type.measure = "mse",
                       nfold = 10,
                       alpha = 0,
                       penalty.facotr = penf 
)

In alasso1_cv I would do something similar, but just replace 1s with the calculated lambda:

penf <- 1 / abs(best_ridge_coef)
penf[1] <- 0 

alasso1_cv <- glmnet(x = x_cont, y = y_cont,
              alpha = 1,
              penalty.factor = penf)

Answer 1

Prediction versus Understanding

Broadly speaking, your goal when fitting a model is either to be able to make predictions, or to understand the relationships between your variables. All kinds of models can be used for both purposes, but some are better suited to prediction (but are hard to understand or interpret), while others are better for understanding (but may make inferior predictions).

Prediction

If your goal is prediction, the distinction between control and non-control variables doesn't make sense. All of your variables are potential predictors, and your goal is to make the best (for a given value of "best") predictions you can with them.

Including a penalty (ridge or lasso) on the terms in your model is usually a good idea because it reduces overfitting, improving your ability to make predictions. That's why penalised regression is mostly used by people interested in prediction.

Does it make sense to include a penalty on some terms but not others? It might, based on the problem at hand, and any prior knowledge you might have. However, the only way to know is to test whether doing so improves the accuracy of your out-of-sample predictions. This is true of all regularisation methods, not just adaptive lasso.

Understanding

If your goal is understanding (as you've said), things are a little different.

First, since your goal is no longer to make predictions, it's not clear what the advantage of using adaptive lasso is, since it's a complex procedure, and not easy to interpret. In fact, standard lasso and ridge regression are both easier to interpret, since they have straightforward Bayesian interpretations, while adaptive lasso doesn't, to my knowledge. Let's say you use the adaptive lasso, with or without setting some penalty terms to zero, and find that it uses one of your variables as a predictor. What can you conclude from this?

There are a few alternative approaches. The simplest is just to use linear regression, and see what predictors are statistically significant (note that correlated predictors pose a problem for all regression models, so that's a parallel issue). Since you're interested in being "conservative", you have a few options. You could use a stricter threshold for significance, for instance only concluding there is an effect if $p < .001$ (see Lakens et al, 2008).

Alternatively, you could fit a Bayesian regression model, setting conservative priors on the parameters for variables you believe should have small effects. In this case, it absolutely makes sense to set a uniform prior on some control variables (equivalent to setting the penalty to 0), but using a more conservative prior for other predictors. You get to use your own judgement for this!

On a practical level, you can use the brms package for R for this (this is a good tutorial on linear models in brms), standardise your predictors and your outcome variable to have a standard deviation of 1, and set, for instance, a $\text{Normal}(0, 1)$ prior (set_prior("normal(0, 1)", class = "b")) to indicate that you're 95% certain the standardised regression weights (change in standard deviations of the outcome per one standard deviation change of the predictor) should be between $-2$ and $+2$.

Obviously, there's a lot of details I'm skipping here. I would suggest Richard McElreath's excellent Statistical Rethinking book as a starting point.