3
$\begingroup$

Lets take this example for how to conduct an adaptive LASSO. Essentially an initial model is fit using ridge regression. Then, a LASSO is fit, in which the value of lambda is tuned individually for each predictor based on its coefficient in the ridge regression.

Suppose I would like to include a control variable.

I know I can alter the model such that lambda doesn't get applied to a certain variable by setting the penalty factor to be 0 for those variables (i.e., the candidate control variables).

My question is: Does it make more sense to do this in both the initial ridge model and the rescale model (i.e., ride1_cv and alasso1_cv in the example). Or just in the rescale model?


Code for setting lambdas to 0:

For example, in ridge1_cv I would:

#create a vector of 0 for my control variable and 1s everywhere else, and add a penalty factor term to the model:

penf <- c(0, rep(1, times = (ncol(d)-1))

ridge1_cv <- cv.glmnet(x = x_cont, y = y_cont,
                       type.measure = "mse",
                       nfold = 10,
                       alpha = 0,
                       penalty.facotr = penf 
)

In alasso1_cv I would do something similar, but just replace 1s with the calculated lambda:

penf <- 1 / abs(best_ridge_coef)
penf[1] <- 0 

alasso1_cv <- glmnet(x = x_cont, y = y_cont,
              alpha = 1,
              penalty.factor = penf)
$\endgroup$
12
  • $\begingroup$ Is your goal to build a predictive model, or to understand the relationship between your predictors and the outcome? If you're interested in prediction, could you please explain the distinction between "control" and "non-control" variables - isn't everything just a predictor? If you're interested in understanding, why use lasso? $\endgroup$
    – Eoin
    Sep 3, 2020 at 8:55
  • $\begingroup$ To understand the relationship. Using lasso because it is conservative and is happy to deal with correlated predictors. I'm also using adaptive lasso which I understand to be better for understanding. $\endgroup$
    – Dave
    Sep 3, 2020 at 14:47
  • $\begingroup$ I think it's fair to say that this is an extremely convoluted way of trying to understand a linear relationship. There are sensible Bayesian interpretations of both ridge regression (a Gaussian prior) and lasso (MAP with Laplace prior). Adaptive lasso is useful, particularly for data-mining, but I have no idea how you could interpret its estimates scientifically, let alone publish them. Basically, my advice is to forget about adaptive lasso, and just use Bayesian linear regression if you want conservative parameter estimates. $\endgroup$
    – Eoin
    Sep 3, 2020 at 18:39
  • $\begingroup$ @Dave, LASSO is not happy to deal with correlated predictors; it is the opposite. Ridge is better suited for dealing with correlated predictors. $\endgroup$ Sep 4, 2020 at 8:22
  • $\begingroup$ @Eoin Thanks, I appreciate the advice! Could you give me a bit more detail on the methods you would suggest using instead? $\endgroup$
    – Dave
    Sep 7, 2020 at 8:31

1 Answer 1

2
$\begingroup$

Prediction versus Understanding

Broadly speaking, your goal when fitting a model is either to be able to make predictions, or to understand the relationships between your variables. All kinds of models can be used for both purposes, but some are better suited to prediction (but are hard to understand or interpret), while others are better for understanding (but may make inferior predictions).

Prediction

If your goal is prediction, the distinction between control and non-control variables doesn't make sense. All of your variables are potential predictors, and your goal is to make the best (for a given value of "best") predictions you can with them.

Including a penalty (ridge or lasso) on the terms in your model is usually a good idea because it reduces overfitting, improving your ability to make predictions. That's why penalised regression is mostly used by people interested in prediction.

Does it make sense to include a penalty on some terms but not others? It might, based on the problem at hand, and any prior knowledge you might have. However, the only way to know is to test whether doing so improves the accuracy of your out-of-sample predictions. This is true of all regularisation methods, not just adaptive lasso.

Understanding

If your goal is understanding (as you've said), things are a little different.

First, since your goal is no longer to make predictions, it's not clear what the advantage of using adaptive lasso is, since it's a complex procedure, and not easy to interpret. In fact, standard lasso and ridge regression are both easier to interpret, since they have straightforward Bayesian interpretations, while adaptive lasso doesn't, to my knowledge. Let's say you use the adaptive lasso, with or without setting some penalty terms to zero, and find that it uses one of your variables as a predictor. What can you conclude from this?

There are a few alternative approaches. The simplest is just to use linear regression, and see what predictors are statistically significant (note that correlated predictors pose a problem for all regression models, so that's a parallel issue). Since you're interested in being "conservative", you have a few options. You could use a stricter threshold for significance, for instance only concluding there is an effect if $p < .001$ (see Lakens et al, 2008).

Alternatively, you could fit a Bayesian regression model, setting conservative priors on the parameters for variables you believe should have small effects. In this case, it absolutely makes sense to set a uniform prior on some control variables (equivalent to setting the penalty to 0), but using a more conservative prior for other predictors. You get to use your own judgement for this!

On a practical level, you can use the brms package for R for this (this is a good tutorial on linear models in brms), standardise your predictors and your outcome variable to have a standard deviation of 1, and set, for instance, a $\text{Normal}(0, 1)$ prior (set_prior("normal(0, 1)", class = "b")) to indicate that you're 95% certain the standardised regression weights (change in standard deviations of the outcome per one standard deviation change of the predictor) should be between $-2$ and $+2$.

Obviously, there's a lot of details I'm skipping here. I would suggest Richard McElreath's excellent Statistical Rethinking book as a starting point.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.