I build a linear regression model and use it to predict out-of-sample. In this context, I use LOOCV and k-fold CV (5). However, both methods seem to lead to the same results. The only minor difference between these two methods are slightly different values for the accuracy measures for the in-sample estimates (see results below).
What is going on here; am I missing a point?
library(mlbench) library(caret) data(BostonHousing) df <- BostonHousing ###### set.seed(12345) train.index <- createDataPartition(df$medv, p = 0.75, list = FALSE) train <- df[train.index, ] test <- df[-train.index, ] ##### fitControl <- trainControl(method = "LOOCV") mod1 <- train(medv ~ crim + zn + rm, data = train, method = "lm", trControl = fitControl) preds1 <- predict(mod1, newdata = test) ##### fitControl2 <- trainControl(method = "repeatedcv", number = 5, repeats = 10) mod2 <- train(medv ~ crim + zn + rm, data = train, method = "lm", trControl = fitControl2) preds2 <- predict(mod2, newdata = test)
The results look as follows:
coef(summary(mod1)) coef(summary(mod2)) LOOCV k-fold (Intercept) -28.74077696 -28.74077696 crim -0.23736504 -0.23736504 zn 0.04259996 0.04259996 rm 8.21720224 8.21720224
mod1$results mod2$results LOOCV k-fold RMSE 6.16378 6.083234 Rsquared 0.5437839 0.5727744 MAE 4.176978 4.174368
postResample(preds1, obs = test$medv) postResample(preds2, obs = test$medv) LOOCV k-fold RMSE 4.1298679 4.1298679 Rsquared 0.5489697 0.5489697 MAE 4.1298679 4.1298679