# LOOCV vs. k-fold CV leads to the same results

I build a linear regression model and use it to predict out-of-sample. In this context, I use LOOCV and k-fold CV (5). However, both methods seem to lead to the same results. The only minor difference between these two methods are slightly different values for the accuracy measures for the in-sample estimates (see results below).

What is going on here; am I missing a point?

library(mlbench)
library(caret)
data(BostonHousing)
df <- BostonHousing

######
set.seed(12345)
train.index <- createDataPartition(df$medv, p = 0.75, list = FALSE) train <- df[train.index, ] test <- df[-train.index, ] ##### fitControl <- trainControl(method = "LOOCV") mod1 <- train(medv ~ crim + zn + rm, data = train, method = "lm", trControl = fitControl) preds1 <- predict(mod1, newdata = test) ##### fitControl2 <- trainControl(method = "repeatedcv", number = 5, repeats = 10) mod2 <- train(medv ~ crim + zn + rm, data = train, method = "lm", trControl = fitControl2) preds2 <- predict(mod2, newdata = test)  The results look as follows: Coefficients: coef(summary(mod1)) coef(summary(mod2)) LOOCV k-fold (Intercept) -28.74077696 -28.74077696 crim -0.23736504 -0.23736504 zn 0.04259996 0.04259996 rm 8.21720224 8.21720224  In-Sample fit: mod1$$results mod2$$results LOOCV k-fold RMSE 6.16378 6.083234 Rsquared 0.5437839 0.5727744 MAE 4.176978 4.174368  Out-of-Sample fit: postResample(preds1, obs = test$$medv) postResample(preds2, obs = test$$medv) LOOCV k-fold RMSE 4.1298679 4.1298679 Rsquared 0.5489697 0.5489697 MAE 4.1298679 4.1298679  ## 1 Answer First of all, the final models (mod1$finalModel and mod1$finalModel) are the same, in your particular case for two reasons: 1. You don't actually tune, you train a single model which is the linear model with intercept = TRUE). The telltale line is the output of print(mod2): Tuning parameter 'intercept' was held constant at a value of TRUE You can also look at the mod2$results:

  intercept     RMSE  Rsquared      MAE    RMSESD RsquaredSD     MAESD
1      TRUE 6.121066 0.5568386 4.187102 0.9087823  0.1089092 0.4691107


Compare this to:

mod3 <- train(medv ~ crim + zn + rm,
data = train,
method = "lm",
tuneGrid = data.frame (intercept = c(FALSE, TRUE)),
trControl = fitControl)
mod3
# [...snip...]
# Resampling results across tuning parameters:
#
#   intercept  RMSE      Rsquared   MAE
#   FALSE      6.818821  0.4592127  4.844369
#    TRUE      6.163780  0.5437839  4.176978
#
# RMSE was used to select the optimal model using the smallest value.
# The final value used for the model was intercept = TRUE.

mod3\$results
#   intercept     RMSE  Rsquared      MAE
# 1     FALSE 6.818821 0.4592127 4.844369
# 2      TRUE 6.163780 0.5437839 4.176978

2. The inner cross validation during the tuning results in a hyperparameter set, but not yet in the final model. The final model is obtained by training with this hyperparameter set and all data that was handed over to train().

Regardless of the cross validation/bootstrap routine you chose, as long as the tuned hyperparameter set is the same, the final model will also be the same (at least for deterministic training routines such as lm()).

So, even if you had tuned, still the same model would result: the one with intercept = TRUE.