I'm using linear regression to predict a continuous variable using a large number (~200) of binary indicator variables. I have around 2,500 data rows. There are a couple of issues here:

  • When I run univariate regressions, most of the indicators have significant associations with the dependent variable (by design - I collected data I thought would be associated with the dependent variable)
  • Some of the indicators have very low incidence (~1% or less) (but I try to exclude these variables from analysis)
  • Some of the indicators co-occur

I'm looking for a stable, parsimonious model. I've been using AIC or BIC selection. I'm aware this can be frowned upon, but I'm comfortable with it, primarily because I'm only feeding in variables I'd expect to have some association. I don't want to use a standard Enter regression because of IDV (independent variable) co-occurrence.

Unfortunately, if I construct a new dataset using random draws of cases from the original dataset, and refit my BIC model, I find that after 20 such loops the majority of the terms each individual BIC model retains are not preserved across all 20 models. I'm thinking I should use some sort of regularized regression - say ridge (for its ability to deal with IDV correlations), lasso (for its ability to set parameters to 0) or elastic net (to combine these advantages). My understanding is that this is akin to using a softer thresholding criteria than stepwise AIC or BIC that can yield more repeatable results. I've never used these techniques before, but after doing some research I'm comfortable selecting a tuning parameter using cross validation to maximise performance in terms of prediction error (though if anyone feels compelled to provide a walkthrough here, I wouldn't object).

My main question: I'm actually more interested in interpreting my model than I am in using it to make predictions. I've read that selecting a tuning parameter for interpretation is harder than it is for prediction, but haven't found any more information. Can someone point me in the correct direction? I'm using R, if anyone is curious.

A secondary question: R provides (I guess Wald) significance estimates for the models in a lasso model. Do I trust them? Can I, at a pinch, interpret them?

I have an engineering background, so I'd particularly like referenced answers (web resources are fine) if possible, and math is welcome as long as its provided with an explanation/intuition.


  • $\begingroup$ (1) model reduction is challenging (2) coefficient estimates (and p-values) from the step-wise methods you've outlined above are likely biased (3) if you'r goal is model interpretation I'm not sure a shrinkage method will be useful (4) nothing is better than a reasonable pre-specified model $\endgroup$
    – charles
    Commented Mar 2, 2014 at 18:41
  • $\begingroup$ (1) Agreed (2) Could you elaborate/point me in the direction of something more formal so I can get a better sense as to why (3) how so - I'm definitely interested in this, can you expand? (4) there is no theory for the application I'm working on, and all IDV's are variables we felt were likely to impact the DV. We want to establish an importance heirachy and deal with the issue of collinearity. $\endgroup$ Commented Mar 2, 2014 at 22:12
  • $\begingroup$ I should mention this paper is often references: arxiv.org/abs/1001.0188. If you bootstrap AIC your models will likely still be variable but more stable (essentially higher p-value cut-off) (2) Steyerberg/Harrell usually quoted on this but think has been done often.(Steyerberg. Stepwise selection in small datasets.... J Clin Epi 1999: 52(10) 935 (3) shrinkage intentionally biases coefficients for better prediction. So then people sometimes compensate by things post lasso estimation (first reference) which is beyond my knowledge base... $\endgroup$
    – charles
    Commented Mar 3, 2014 at 1:27
  • $\begingroup$ This may also be of interest: andrewgelman.com/2013/03/18/… and stats.stackexchange.com/questions/34859/… $\endgroup$
    – charles
    Commented Mar 4, 2014 at 3:01

2 Answers 2


I'd like to try answer your main question, here are two options:

  1. Use the one-standard-error (1SE) rule

    • When cross-validating for selection purposes, it can help to use the 1SE rule. The standard error of the CV estimate is calculated for each fold. Instead of selecting the model corresponding to the minimum CV error, use the most parsimonious model where the CV error is within one standard error of the minimum.

    • The R package glmnet does this automatically - select lambda.1se instead of lambda.min.

    • The 1SE rule can help select the correct model using LASSO but the estimates could suffer large bias. Usually a large tuning parameter is necessary to set coefficients to zero (especially if there are many that are zero). LASSO shrinks coefficients equally, so the large tuning parameter can overshrink large coefficients and this causes the large bias. Shrinkage is supposed to introduce slight bias but with a substantial decrease in variance, which causes lower prediction error. In this case, the bias is large!

    • Edit: Some authors propose overcoming the bias by using the LASSO for variable selection and then estimating the parameters of the selected subset using least squares - see for example the LARS-OLS Hybrid, or thresholding the LASSO discussed by Bühlmann and van de Geer (2011)
  2. Use the adaptive LASSO or relaxed LASSO

    • The LASSO often includes too many variables when selecting the tuning parameter for prediction (minimum CV error). But, with high probability, the true model is a subset of these variables. This suggests using a secondary stage of estimation. The adaptive LASSO and relaxed LASSO both achieve this and control the bias of the LASSO estimate. Using either method, the prediction-optimal tuning parameter leads to consistent selection.
    • The R package relaxo implements relaxed LASSO.
    • For adaptive LASSO, you need an initial estimate, either least squares, ridge or even LASSO estimates, to calculate weights. Then you can implement it using the same function that you would for LASSO by scaling the X matrix:

      w  <- abs(beta.init)  
      x2 <- scale(x, center=FALSE, scale=1/w)  

      Select tuning parameter and estimate coefficients (coef) using x2

      coef <- coef*w
  3. Edit: I've come across a few other criteria which can be used for variable selection with the LASSO:

    • Wang, et al (2009) proposed a modified BIC criterion and show that it is consistent. Basically a factor $C_{n}$ is multiplied to the BIC penalty. They chose $C_{n}=log(log(p))$ when $p$ varies with $n$. For fixed $p$, Chand (2012) showed consistency using $C_{n}=\sqrt n/p$. Fan and Tang (2013) proposed a generalized version for use when $p>n$.

    • Roberts and Nowak (2014) propose using percentile CV, repeatedly performing cross-validation to yield a vector of tuning parameter estimates and then selecting the optimal one as the 95% percentile of that vector.

    • Sun, et al (2013) propose using Cohen's kappa coefficient which measures the agreement between two sets. Fang, et al (2013) take the ratio of the average kappa coefficient and the average CV error, which they call PASS. These two methods can be implemented using the R package pass.

In my conference paper I have done simulations studies to compare the performance of tuning parameter selection methods for the LASSO, both for prediction and variable selection purposes. For variable selection, I implemented the 1SE rule with 5 fold- and 10 fold CV, percentile CV, kappa, PASS, BIC and the modified BIC. The results show that most of these methods do perform better variable selection than prediction methods such as k fold CV, LOOCV, GCV, Cp and AIC.

  • $\begingroup$ Go ahead & try again, @StatGrrl, you may be able to post more links now. Even if not, they will show up in the post (just as not clickable, I think), & a higher-rep user will be able to edit them in. Welcome to our site! Since you are new here, you may want to take our tour, which has info for new users. $\endgroup$ Commented Jul 19, 2014 at 13:59
  • $\begingroup$ StatGrrl, when using adaptive LASSO and the "x2" matrix you described, is it still required to use the "standardize=TRUE" option in glmnet ? $\endgroup$
    – yoyo
    Commented Aug 19, 2015 at 9:42
  • $\begingroup$ @yoyo, I don't believe so. According to Zou's 2006 paper, the data is only centered, not standardized. $\endgroup$
    – StatGrrl
    Commented Aug 20, 2015 at 11:04
  • $\begingroup$ @yoyo, I'm just reconsidering your question... when applying glmnet to the x2 matrix then do not standardize. But it may still be best to standardize x before applying least squares. For example, see this R function $\endgroup$
    – StatGrrl
    Commented Aug 20, 2015 at 11:21

Not sure whether this helps, but potentially you can create (using e.g. the caret package) a plot akin How to interpret the lasso selection plot , and subsequently choose a cross-validated tuning value that restricts the parameter space sufficiently to allow interpretability?

  • $\begingroup$ This was my original plan, however, I get the impression that there may be different (formal) criteria for 'interpretability' as opposed to 'prediction error'. Unhelpfully, I can't find the source for this I encountered while trawling the net. $\endgroup$ Commented Mar 2, 2014 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.