9
$\begingroup$

I understand that Stepwise regression analysis has lots of limitations, including the assumption that the predictors are not highly correlated with each other. In fact, this limitation was the most important reason that I switched to Elastic Net, as I had 75 predictors in my model, some of which are highly correlated.

Using Elastic Net, I could reduce my predictors to 21. I used these selected 21 variables in a multilinear regression model and calculated the coefficient of determination ($R^2=0.58$).

However, when I used Stepwise analysis on the same data, only 11 variables got selected, while the R-square stayed the same! Does it mean that my results from Stepwise analysis can explain a higher proportion of my outcome? If so, how can I justify the limitations of Stepwise analysis over Elastic Net when I'm getting better results?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ It is not appropriate to enter variables found by lasso or elastic net into a new regression as this ignores shrinkage. And stepwise regression without shrinkage is invalid. $\endgroup$
    – Frank Harrell
    Commented Sep 10, 2013 at 12:04
  • $\begingroup$ Prof @Frank Harrell, I wonder if you could comment further on this statement- there appears to be studies using elastic net regularization as a variable selection method, and then performing PLS regression on the subset of variables. $\endgroup$
    – hatmatrix
    Commented Nov 3, 2014 at 7:01
  • $\begingroup$ I don't understand how anyone would thank that double dipping is valid. The second step forgets about the shrinkage of the first. $\endgroup$
    – Frank Harrell
    Commented Nov 3, 2014 at 15:34
  • $\begingroup$ Thanks for your answer Prof @Frank Harrell. As for the double-dipping, is it necessary that the shrinkage be consistent with the selection of latent variables? I see that there is a discordancy in the criteria for model selection, but in the end a prediction model constructed by separation of the variable selection and latent variable selection can still be a "valid" model? $\endgroup$
    – hatmatrix
    Commented Nov 5, 2014 at 4:44
  • $\begingroup$ Latent variable has a special meaning that I don't think applies here. Variable selection is frought with difficulties. The only thing that makes it work is the simultaneous incorporation of shrinkage. You cannot separate the two. An alternate approach that preserves the correct amount of shrinkage is pre conditioning (also called model approximation) whereby the linear predictor ($\hat{Y}$} from lasso or elastic net or any other method is predicted from individual predictors in an attempt to simplify the properly shrunken model. $\endgroup$
    – Frank Harrell
    Commented Nov 5, 2014 at 13:14

1 Answer 1

Reset to default
11
$\begingroup$

Your question has an implicit assumption that $R^2$ is a good measure of the quality of the fit and is appropriate for comparing between models. I think that your background information provides evidence that $R^2$ is not a good tool for what you are trying to do. After all, you can increase $R^2$ by adding nonsense variables to your model.

Did you take the variables that were found using the elastic net and refit a new regression model using those variables rather than use the estimates from the elasticnet fit? That is kind of like entering your data into a nice statistical software program and using it to round the data and print it out so you can calculate the mean using an abacus.

If you want the fewest predictors possible (and still get a reasonable fit) then lasso methods will tend to result in fewer predictors than elasticnet methods. The advantage of the elasticnet method is not in finding the fewest variables, but in finding a good model that takes advantage of the information in the variables and avoids the bias that you get with stepwise models.

A better comparison would be how well they predict a new set of observations, or maybe a press statistic or cross-validation.

Improve this answer
$\endgroup$
5
  • $\begingroup$ generalised double pareto is better than lasso, as it doesn't shrink the "clearly" non zero betas. Horseshoe is another. $\endgroup$
    – probabilityislogic
    Commented Oct 20, 2012 at 6:03
  • $\begingroup$ Hi @probabilityislogic have you got a link comparing generalized double pareto to lasso? And horseshoe? $\endgroup$
    – Peter Flom
    Commented Oct 20, 2012 at 12:55
  • $\begingroup$ @GregSnow: Thanks so much for you response. I do agree that R^2 is not the right measure to compare the models. However, since that I don't have another set of data, what do you think is the best measure to compare these two models? I was thinking of MSE, but again, I have to get the selected variables from the two models, make a regression, and find the MSE for both, right? Isn't it again using an abacus after using a nice software? $\endgroup$
    – Niousha
    Commented Oct 20, 2012 at 21:11
  • $\begingroup$ @GregSnow: Also, my dataset is not very large (n=90). That's why I doubt separating the data into test and train sets. So, I would be grateful if I can find another measure for comparison. $\endgroup$
    – Niousha
    Commented Oct 20, 2012 at 22:14
  • 1
    $\begingroup$ here is a link which compares the shrinkage properties of the lasso and horseshoe. Similar results would occur for the generalised double pareto, as it has similar tail behaviour. It has a lot to do with local-global shrinkage rules. Lasso is just a global shrinkage rule, so it has real difficulty when you have both sparsity and large signals. I'm pretty sure this is known via the inconsistent parameter estimates for non-zero betas. $\endgroup$
    – probabilityislogic
    Commented Oct 21, 2012 at 5:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.