I would like to pose this question in two parts. Both deal with a generalized linear model, but the first deals with model selection and the other deals with regularization.

Background: I utilize GLMs (linear, logistic, gamma regression) models for both prediction and for description. When I refer to the "normal things one does with a regression" I largely mean description with (i) confidence intervals around coefficients, (ii) confidence intervals around predictions and (iii) hypothesis tests concerning linear combinations of the coefficients such as "is there a difference between treatment A and treatment B?".

Do you legitimately lose the ability to do these things using the normal theory under each of the following? And if so, are these things really only good for models used for pure prediction?

I. When a GLM has been fit via some model selection process (for concreteness say its a stepwise procedure based on AIC).

II. When a GLM has been fit via a regularization method (say using glmnet in R).

My sense is that for I. the answer is technically that you should use a bootstrap for the "normal things one does with a regression", but no one really abides by that.

After getting a few responses and reading elsewhere, here is my take on this (for anyone else benefit as well as to receive correction).

A) RE: Error Generalize. In order to generalize error rates on new data, when there is no hold out set, cross validation can work but you need to repeat the process completely for each fold - using nested loops - thus any feature selection, parameter tuning, etc. must be done independently each time. This idea should hold for any modeling effort (including penalized methods).

B) RE: Hypothesis testing and confidence intervals of GLM. When using model selection (feature selection, parameter tuning, variable selection) for a generalized linear model and a hold out set exists, it is permissible to train the model on a partition and then fit the model on the remaining data or the full data set and use that model/data to perform hypothesis tests etc. If a hold out set does not exist, a bootstrap can be used, as long as the full process is repeated for each bootstrap sample. This limits the hypothesis tests that can be done though since perhaps a variable will not always be selected for example.

C) RE: Not carrying about prediction on future data sets, then fit a purposeful model guided by theory and a few hypothesis tests and even consider leaving all variables in the model (significant or not) (along the lines of Hosmer and Lemeshow). This is small variable set classical type of regression modeling and then allows the use of CI's and hypothesis test.

D) RE: Penalized regression. No advice, perhaps consider this suitable for prediction only (or as a type of feature selection to then apply to another data set as in B above) as the bias introduced makes CI's and hypothesis tests unwise - even with the bootstrap.

  • 1
    $\begingroup$ People sometimes do this - unknowingly (ie misuse Statistics, because they get the desired result) and knowingly (they did bootstrap and it did not affect the result substantially). Your point is valid, and Professor Harrell points this out in the Preface of his book that bootstrap is beneficial. $\endgroup$
    – suncoolsu
    Commented Feb 15, 2011 at 5:19
  • 1
    $\begingroup$ Here is something like "yes" for your point (II): arxiv.org/abs/1001.0188 $\endgroup$
    – Alex
    Commented Jul 26, 2011 at 21:54

2 Answers 2


You might check out David Freedman's paper, "A Note on Screening Regression Equations." (ungated)

Using completely uncorrelated data in a simulation, he shows that, if there are many predictors relative to the number of observations, then a standard screening procedure will produce a final regression that contains many (more than by chance) significant predictors and a highly significant F statistic. The final model suggests that it is effective at predicting the outcome, but this success is spurious. He also illustrates these results using asymptotic calculations. Suggested solutions include screening on a sample and assessing the model on the full data set and using at least an order of magnitude more observations than predictors.

  • $\begingroup$ Note: For the bootstrap to be an effective solution, you'd have to bootstrap the entire procedure, starting before any screening occurs, screen the bootstrapped sample, then calculate coefficients. But now you have different sets in predictors in each regression and it's no longer clear how to calculate the distribution for any one of them. Bootstrapping confidence intervals for predicted values of the outcome may be effective, however. $\endgroup$
    – Charlie
    Commented Feb 15, 2011 at 3:44
  • $\begingroup$ @charlie: [Do I read you correctly that you you are only speaking to I.(model selection) not II. (penalized)] Are you saying that for prediction intervals, it is valid to use model selection and then bootstrap the predictions from that model, but for anything else you need to bootstrap the entire process? $\endgroup$
    – B_Miner
    Commented Feb 16, 2011 at 1:53
  • $\begingroup$ @charlie Regarding the suggested solution of screening on a sample. Would that be along the lines of partitioning the data, (ab)using one set (model selection etc) and then applying that model to the remaining data - and on that data with the model that was fit using traditional theory for hypothesis tests, CIs etc? $\endgroup$
    – B_Miner
    Commented Feb 16, 2011 at 2:03
  • $\begingroup$ I was thinking only of model selection, but that's largely because I don't know all that much about penalized regression. I would say that you need to bootstrap the entire process in order to get inference on predictions from the model. The whole issue is that, in any one sample, you are likely to find spurious correlations that get magnified when you include some variables and leave others out. The only way to get around this is to look at multiple samples---i.e., bootstrap. Of course, no one actually does this. $\endgroup$
    – Charlie
    Commented Feb 16, 2011 at 15:55
  • $\begingroup$ Right, you use one partition of your sample to come up with your model using model selection procedures, then do your inference on either the other partition or the full sample. $\endgroup$
    – Charlie
    Commented Feb 16, 2011 at 15:59

Regarding 1) Yes, you do lose this. See e.g. Harrell Regression Modeling Strategies, a book published by Wiley or a paper I presented with David Cassell called "Stopping Stepwise" available e.g. www.nesug.org/proceedings/nesug07/sa/sa07.pdf

  • $\begingroup$ I have seen this paper - very interesting. Two questions. 1) Lets take logistic regression. It sounds like the only way to conduct CI or hypothesis tests is to build a model in the style of hosmer and lemeshow (precluding any data sets with big p)? So you are left with "using" the model for only point estimates? 2) Your paper discusses the lasso among other alternatives. Are you of the mind that this allows later hypothesis testing or is "simply" given as a better option of model selection? $\endgroup$
    – B_Miner
    Commented Feb 16, 2011 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.