# use Cross-validation to find the better model

train.control <- trainControl(method = "LOOCV")
model_1 <- train(end_confidence_6 ~ SC0 + Review_notes + Ask_qs + Perfect_attendance + Post_attendance + Discussion, data = df_lm, method = "lm", trControl = train.control)
model_2 <- train(end_confidence_6 ~ SC0 + Review_notes + Ask_qs + Perfect_attendance, data = df_lm, method = "lm", trControl = train.control)

criterias <- rbind(model_1$results,model_2$results)
Index_2 <- c(which.max(criterias$Rsquared), which.min(criterias$RMSE))
Criteria_2 <- c("max_Rsquared","min_RMSE")
from_cross_validation <- data.frame(Criteria_2, Index_2)
from_cross_validation

Criteria_2 Index_2
1 max_Rsquared       2
2     min_RMSE       2


Based on the result of LOOCV, the model_2 is better than model_1. However, when I saw the summary of model_1 and model_2, I found that the Adjusted R-squared of model_1 is greater than model_2's, and the Residual standard error of model_1 is smaller than model_2's, which indicate that the model_1 is the better. I am confused about why it happens. And which criteria are more powerful to decide the best predictive model in general case? Thank you for any response!

summary(model_1)

Call:
lm(formula = .outcome ~ ., data = dat)

Residuals:
Min      1Q  Median      3Q     Max
-1.0792 -0.3171  0.0282  0.3063  1.0312

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)         4.57006    0.48580   9.407 4.09e-11 ***
SC0                 0.19239    0.07801   2.466  0.01870 *
Review_notes       -0.31053    0.10298  -3.015  0.00475 **
Ask_qs              0.27789    0.12130   2.291  0.02810 *
Perfect_attendance -0.29209    0.11396  -2.563  0.01483 *
Post_attendance1    0.23294    0.19991   1.165  0.25183
Discussion          0.13839    0.09869   1.402  0.16967
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.5586 on 35 degrees of freedom
Multiple R-squared:  0.5062,    Adjusted R-squared:  0.4216
F-statistic: 5.981 on 6 and 35 DF,  p-value: 0.0002224

summary(model_2)
Call:
lm(formula = .outcome ~ ., data = dat)

Residuals:
Min       1Q   Median       3Q      Max
-1.29148 -0.30765  0.00989  0.28818  1.09152

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)         4.88253    0.45034  10.842 4.84e-13 ***
SC0                 0.17326    0.07502   2.309  0.02661 *
Review_notes       -0.32484    0.09531  -3.408  0.00159 **
Ask_qs              0.35021    0.11457   3.057  0.00414 **
Perfect_attendance -0.24511    0.11007  -2.227  0.03212 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.5647 on 37 degrees of freedom
Multiple R-squared:  0.4667,    Adjusted R-squared:  0.409
F-statistic: 8.094 on 4 and 37 DF,  p-value: 8.576e-05

• Did you test the models? (That can be really important; although you did use CV.) As far as error versus R squared—they are not inverses of each other. If you have all of the points surrounding the line of fit, but none really on it, you might have a lower error and higher R-squared than if you had a wider range of distances, but with many points on the line of fit. (It doesn't fit as well overall, but the error is lower.) The differences in error are really negligible, though. When it comes to error how much data you have plays a large role here, as well. How is the homogeneity looking?
– Kat
Jun 11 at 16:41
• @Kat Thank you for the idea! My goal is to select the variables and find the best predictive model. And I got model_1 and model_2 from regsubsets() function, since they maximize the adj R^2 and minimize the Mallow's Cp. What do you mean by "test the model" ? test it on a new dataset? Jun 11 at 18:08
• Typically, you would split the data you're going to model. You use some to train the model and the remainder to test the model. This is the easiest way to test the model's performance and to assess for overfitting.
– Kat
Jun 12 at 7:08