0
$\begingroup$ 
train.control <- trainControl(method = "LOOCV")
model_1 <- train(end_confidence_6 ~ SC0 + Review_notes + Ask_qs + Perfect_attendance + Post_attendance + Discussion, data = df_lm, method = "lm", trControl = train.control)
model_2 <- train(end_confidence_6 ~ SC0 + Review_notes + Ask_qs + Perfect_attendance, data = df_lm, method = "lm", trControl = train.control)

criterias <- rbind(model_1$results,model_2$results)
Index_2 <- c(which.max(criterias$Rsquared), which.min(criterias$RMSE))
Criteria_2 <- c("max_Rsquared","min_RMSE")
from_cross_validation <- data.frame(Criteria_2, Index_2)
from_cross_validation

     Criteria_2 Index_2
 1 max_Rsquared       2
 2     min_RMSE       2

Based on the result of LOOCV, the model_2 is better than model_1. However, when I saw the summary of model_1 and model_2, I found that the Adjusted R-squared of model_1 is greater than model_2's, and the Residual standard error of model_1 is smaller than model_2's, which indicate that the model_1 is the better. I am confused about why it happens. And which criteria are more powerful to decide the best predictive model in general case? Thank you for any response!

summary(model_1)

Call:
lm(formula = .outcome ~ ., data = dat)

Residuals:
    Min      1Q  Median      3Q     Max 
-1.0792 -0.3171  0.0282  0.3063  1.0312 

Coefficients:
                   Estimate Std. Error t value Pr(>|t|)    
(Intercept)         4.57006    0.48580   9.407 4.09e-11 ***
SC0                 0.19239    0.07801   2.466  0.01870 *  
Review_notes       -0.31053    0.10298  -3.015  0.00475 ** 
Ask_qs              0.27789    0.12130   2.291  0.02810 *  
Perfect_attendance -0.29209    0.11396  -2.563  0.01483 *  
Post_attendance1    0.23294    0.19991   1.165  0.25183    
Discussion          0.13839    0.09869   1.402  0.16967    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.5586 on 35 degrees of freedom
Multiple R-squared:  0.5062,    Adjusted R-squared:  0.4216 
F-statistic: 5.981 on 6 and 35 DF,  p-value: 0.0002224


summary(model_2)
Call:
lm(formula = .outcome ~ ., data = dat)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.29148 -0.30765  0.00989  0.28818  1.09152 

Coefficients:
                   Estimate Std. Error t value Pr(>|t|)    
(Intercept)         4.88253    0.45034  10.842 4.84e-13 ***
SC0                 0.17326    0.07502   2.309  0.02661 *  
Review_notes       -0.32484    0.09531  -3.408  0.00159 ** 
Ask_qs              0.35021    0.11457   3.057  0.00414 ** 
Perfect_attendance -0.24511    0.11007  -2.227  0.03212 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.5647 on 37 degrees of freedom
Multiple R-squared:  0.4667,    Adjusted R-squared:  0.409 
F-statistic: 8.094 on 4 and 37 DF,  p-value: 8.576e-05
Improve this question
$\endgroup$
3
  • $\begingroup$ Did you test the models? (That can be really important; although you did use CV.) As far as error versus R squared—they are not inverses of each other. If you have all of the points surrounding the line of fit, but none really on it, you might have a lower error and higher R-squared than if you had a wider range of distances, but with many points on the line of fit. (It doesn't fit as well overall, but the error is lower.) The differences in error are really negligible, though. When it comes to error how much data you have plays a large role here, as well. How is the homogeneity looking? $\endgroup$
    – Kat
    Jun 11 at 16:41
  • $\begingroup$ @Kat Thank you for the idea! My goal is to select the variables and find the best predictive model. And I got model_1 and model_2 from regsubsets() function, since they maximize the adj R^2 and minimize the Mallow's Cp. What do you mean by "test the model" ? test it on a new dataset? $\endgroup$
    – Zheng Astra
    Jun 11 at 18:08
  • $\begingroup$ Typically, you would split the data you're going to model. You use some to train the model and the remainder to test the model. This is the easiest way to test the model's performance and to assess for overfitting. $\endgroup$
    – Kat
    Jun 12 at 7:08

1 Answer 1

Reset to default
0
$\begingroup$

What bothers me the most about the first model is that you have two variables (Post_attendance1, Discussion) with a very high p-value. You should remove them, and then, you basically get the same model (2).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the help! What confused me is that in model_1 the adjusted R^2 is larger than model_2 and the residual standard error is smaller than model_2. Does it mean model_1 is better? $\endgroup$
    – Zheng Astra
    Jun 11 at 18:14
  • $\begingroup$ No, because the two variables interrupt it. It only makes sense that the R^2 is higher (you have more variables), but, in your case, it doesn't mean that the model is better. $\endgroup$
    – Cateded Ur
    Jun 12 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.