I am using R commander to do stepwise model selection in a linear model. When I run stepwise model selection, it reduces some variables, and finally, a model with AIC is provided. However, it does not show the R-squared. I am wondering how I can determine the new value of R-squared? Moreover, is it possible to do stepwise model selection based on adjusted R-squared, not AIC nor BIC?
$\begingroup$
$\endgroup$
3
-
2$\begingroup$ Stepwise regression is fraught with problems yet is common enough that I linked that exact same page a minute ago. If you have to use stepwise regression, you can calculate $R^2$ by using the usual equation: $R^2 = 1-\dfrac{\sum (y_i - \hat y_i)^2}{\sum (y_i - \bar y)^2}$. // Stepwise regression based on adjusted $R^2$ retains the issues discussed in the link. $\endgroup$– DaveCommented Nov 1, 2021 at 17:43
-
$\begingroup$ Thank you Dave. Could you please let me know what command can I use to determine the R-squared based on the results of R after stepwise model selection? $\endgroup$– AminCommented Nov 1, 2021 at 18:57
-
$\begingroup$ I don’t know, but it’s probably something like summary(model)$r.squared. If that does not work, it is easy to code yourself from the equation. $\endgroup$– DaveCommented Nov 1, 2021 at 18:59
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
8
A number of summary statistics of a linear model are not stored in the returned object of class lm, but can be extracted from it with the summary() method:
> fit <- lm(waiting ~ eruptions, faithful)
> names(summary(fit))
[1] "call" "terms" "residuals" "coefficients"
[5] "aliased" "sigma" "df" "r.squared"
[9] "adj.r.squared" "fstatistic" "cov.unscaled"
Adjusted R squared is stored in adj.r.squared. Beware that adjusted R squared is still an in sample estimate, i.e., the data used for fitting the model are also used for evaluation and it is thus still prone to overfitting when used as a model selection cirterion. To compute the out-of-sample R squared, you can fit with glm instead of lm (family=gaussian is equivalent to lm, although the optimization algorithm is different) and use the function cv.glm() from the package boot.
And now a comment for collecting downvotes ;-) There are problems with stepwise selection and it is possible to construct cases where it fails. On the other hand, it is much better than its reputation: an extended study by Hastie and Tibshirani found it to be comparable to an exhaustive full optimal search "throughout", which is quite amazing considering the strong advises against stepwise selection here on CV.
Hastie, Trevor, Robert Tibshirani, and Ryan J. Tibshirani. "Extended comparisons of best subset selection, forward stepwise selection, and the lasso." arXiv:1707.08692 (2017).
-
$\begingroup$ Thank you so much for the detail and also for sharing the paper. $\endgroup$– AminCommented Nov 1, 2021 at 20:37
-
1$\begingroup$ Objections to stepwise selection are as much about a host of misconceptions commonly associated with its employment as about its predictive performance. But if only predictive performance is at issue, the proof of the pudding is in the eating - 'no free lunch' theorems show that no method dominates any other across all classes of problems. Even in his seminal LASSO paper Tibshirani notes that best subset selection performs better given a "small number of large effects". So you're right to suggest stepwise methods shouldn't be, in all circumstances. dismissed out of hand. $\endgroup$– Scortchi ♦Commented Nov 2, 2021 at 12:25
-
1$\begingroup$ @cdalitz: Sorry, our last comments crossed - my last one wasn't a response to yours. My reading of the question is that it's asking how to estimate $R^2$ for the final model, the one left after the stepwise selection. Yes, you can use cross-validated $R^2$ as a criterion for selecting a model, but for estimating out-of-sample performance you need an outer cross-validation loop that repeats the whole estimation procedure, including the stepwise selection (which is estimating some coefficients to be zero). If you're holding out data rather than cross-validating, the equivalent is ... $\endgroup$– Scortchi ♦Commented Nov 2, 2021 at 12:45
-
1$\begingroup$ ... the standard (in Machine Learning) training - validation - test partitioning. See e.g. Feature selection and cross-validation. $\endgroup$– Scortchi ♦Commented Nov 2, 2021 at 12:51
-
1$\begingroup$ @scortchi-reinstate-monica I see your point. Actually, this warning applies more generally: estimating a performance index from a model that has been optimized on basis of exactly the same performance index will result in an optimistic bias. $\endgroup$– cdalitzCommented Nov 2, 2021 at 12:51
$\begingroup$
$\endgroup$
1
I'm not familiar with R commander but I assume you can re-fit the final model to get various statistics including R-squared. Selecting on AIC is very similar to adjusted R-Square but includes a bias correction so I would stick with it. The most important part however is that selecting a model based on stepwise regression is a bad idea.
-
$\begingroup$ Note that the upvoted answers in the link advocate against stepwise procedures and that the only answer advocating in favor of stepwise procedures has a -57 score. $\endgroup$– DaveCommented Nov 1, 2021 at 17:46