How to select tuning parameter for regularized regressions for interpretation?

Question

I'm using linear regression to predict a continuous variable using a large number (~200) of binary indicator variables. I have around 2,500 data rows. There are a couple of issues here:

• When I run univariate regressions, most of the indicators have significant associations with the dependent variable (by design - I collected data I thought would be associated with the dependent variable)
• Some of the indicators have very low incidence (~1% or less) (but I try to exclude these variables from analysis)
• Some of the indicators co-occur

I'm looking for a stable, parsimonious model. I've been using AIC or BIC selection. I'm aware this can be frowned upon, but I'm comfortable with it, primarily because I'm only feeding in variables I'd expect to have some association. I don't want to use a standard Enter regression because of IDV (independent variable) co-occurrence.

Unfortunately, if I construct a new dataset using random draws of cases from the original dataset, and refit my BIC model, I find that after 20 such loops the majority of the terms each individual BIC model retains are not preserved across all 20 models. I'm thinking I should use some sort of regularized regression - say ridge (for its ability to deal with IDV correlations), lasso (for its ability to set parameters to 0) or elastic net (to combine these advantages). My understanding is that this is akin to using a softer thresholding criteria than stepwise AIC or BIC that can yield more repeatable results. I've never used these techniques before, but after doing some research I'm comfortable selecting a tuning parameter using cross validation to maximise performance in terms of prediction error (though if anyone feels compelled to provide a walkthrough here, I wouldn't object).

My main question: I'm actually more interested in interpreting my model than I am in using it to make predictions. I've read that selecting a tuning parameter for interpretation is harder than it is for prediction, but haven't found any more information. Can someone point me in the correct direction? I'm using R, if anyone is curious.

A secondary question: R provides (I guess Wald) significance estimates for the models in a lasso model. Do I trust them? Can I, at a pinch, interpret them?

I have an engineering background, so I'd particularly like referenced answers (web resources are fine) if possible, and math is welcome as long as its provided with an explanation/intuition.

Thanks!

Answer 1

I'd like to try answer your main question, here are two options:

1. Use the one-standard-error (1SE) rule

   • When cross-validating for selection purposes, it can help to use the 1SE rule. The standard error of the CV estimate is calculated for each fold. Instead of selecting the model corresponding to the minimum CV error, use the most parsimonious model where the CV error is within one standard error of the minimum.

   • The R package glmnet does this automatically - select lambda.1se instead of lambda.min.

   • The 1SE rule can help select the correct model using LASSO but the estimates could suffer large bias. Usually a large tuning parameter is necessary to set coefficients to zero (especially if there are many that are zero). LASSO shrinks coefficients equally, so the large tuning parameter can overshrink large coefficients and this causes the large bias. Shrinkage is supposed to introduce slight bias but with a substantial decrease in variance, which causes lower prediction error. In this case, the bias is large!

   • Edit: Some authors propose overcoming the bias by using the LASSO for variable selection and then estimating the parameters of the selected subset using least squares - see for example the LARS-OLS Hybrid, or thresholding the LASSO discussed by Bühlmann and van de Geer (2011)

2. Use the adaptive LASSO or relaxed LASSO

   • The LASSO often includes too many variables when selecting the tuning parameter for prediction (minimum CV error). But, with high probability, the true model is a subset of these variables. This suggests using a secondary stage of estimation. The adaptive LASSO and relaxed LASSO both achieve this and control the bias of the LASSO estimate. Using either method, the prediction-optimal tuning parameter leads to consistent selection.
   • The R package relaxo implements relaxed LASSO.

   • For adaptive LASSO, you need an initial estimate, either least squares, ridge or even LASSO estimates, to calculate weights. Then you can implement it using the same function that you would for LASSO by scaling the X matrix:

     w  <- abs(beta.init)  
     x2 <- scale(x, center=FALSE, scale=1/w)  
     

     Select tuning parameter and estimate coefficients (coef) using x2

     coef <- coef*w
     

3. Edit: I've come across a few other criteria which can be used for variable selection with the LASSO:

   • Wang, et al (2009) proposed a modified BIC criterion and show that it is consistent. Basically a factor $C_{n}$ is multiplied to the BIC penalty. They chose $C_{n}=log(log(p))$ when $p$ varies with $n$. For fixed $p$, Chand (2012) showed consistency using $C_{n}=\sqrt n/p$. Fan and Tang (2013) proposed a generalized version for use when $p>n$.

   • Roberts and Nowak (2014) propose using percentile CV, repeatedly performing cross-validation to yield a vector of tuning parameter estimates and then selecting the optimal one as the 95% percentile of that vector.

   • Sun, et al (2013) propose using Cohen's kappa coefficient which measures the agreement between two sets. Fang, et al (2013) take the ratio of the average kappa coefficient and the average CV error, which they call PASS. These two methods can be implemented using the R package pass.

In my conference paper I have done simulations studies to compare the performance of tuning parameter selection methods for the LASSO, both for prediction and variable selection purposes. For variable selection, I implemented the 1SE rule with 5 fold- and 10 fold CV, percentile CV, kappa, PASS, BIC and the modified BIC. The results show that most of these methods do perform better variable selection than prediction methods such as k fold CV, LOOCV, GCV, Cp and AIC.

Answer 2

Not sure whether this helps, but potentially you can create (using e.g. the caret package) a plot akin How to interpret the lasso selection plot , and subsequently choose a cross-validated tuning value that restricts the parameter space sufficiently to allow interpretability?