1
$\begingroup$

I build a linear regression model and use it to predict out-of-sample. In this context, I use LOOCV and k-fold CV (5). However, both methods seem to lead to the same results. The only minor difference between these two methods are slightly different values for the accuracy measures for the in-sample estimates (see results below).

What is going on here; am I missing a point?

library(mlbench)
library(caret)
data(BostonHousing)
df <- BostonHousing

######
set.seed(12345)
train.index <- createDataPartition(df$medv, p = 0.75, list = FALSE)
train <- df[train.index, ]
test <- df[-train.index, ]

#####
fitControl <- trainControl(method = "LOOCV")

mod1 <- train(medv ~ crim + zn + rm,
              data = train,
              method = "lm",
              trControl = fitControl)

preds1 <- predict(mod1, newdata = test)

#####
fitControl2 <- trainControl(method = "repeatedcv", number = 5, repeats = 10)

mod2 <- train(medv ~ crim + zn + rm,
              data = train,
              method = "lm",
              trControl = fitControl2)

preds2 <- predict(mod2, newdata = test)

The results look as follows:

Coefficients:

coef(summary(mod1)) 
coef(summary(mod2))

             LOOCV         k-fold
(Intercept) -28.74077696  -28.74077696
crim         -0.23736504   -0.23736504
zn            0.04259996    0.04259996
rm            8.21720224    8.21720224

In-Sample fit:

mod1$results 
mod2$results

              LOOCV         k-fold
RMSE          6.16378       6.083234
Rsquared      0.5437839     0.5727744
MAE           4.176978      4.174368

Out-of-Sample fit:

postResample(preds1, obs = test$medv)
postResample(preds2, obs = test$medv)

              LOOCV         k-fold
RMSE          4.1298679     4.1298679
Rsquared      0.5489697     0.5489697
MAE           4.1298679     4.1298679
$\endgroup$
1
$\begingroup$

First of all, the final models (mod1$finalModel and mod1$finalModel) are the same, in your particular case for two reasons:

  1. You don't actually tune, you train a single model which is the linear model with intercept = TRUE).

    The telltale line is the output of print(mod2):

    Tuning parameter 'intercept' was held constant at a value of TRUE

    You can also look at the mod2$results:

      intercept     RMSE  Rsquared      MAE    RMSESD RsquaredSD     MAESD  
    1      TRUE 6.121066 0.5568386 4.187102 0.9087823  0.1089092 0.4691107
    

    Compare this to:

    mod3 <- train(medv ~ crim + zn + rm,
               data = train,
               method = "lm",
               tuneGrid = data.frame (intercept = c(FALSE, TRUE)),
               trControl = fitControl)
    mod3
    # [...snip...]
    # Resampling results across tuning parameters:
    # 
    #   intercept  RMSE      Rsquared   MAE     
    #   FALSE      6.818821  0.4592127  4.844369
    #    TRUE      6.163780  0.5437839  4.176978
    # 
    # RMSE was used to select the optimal model using the smallest value.
    # The final value used for the model was intercept = TRUE.
    
    mod3$results
    #   intercept     RMSE  Rsquared      MAE
    # 1     FALSE 6.818821 0.4592127 4.844369
    # 2      TRUE 6.163780 0.5437839 4.176978
    
  2. The inner cross validation during the tuning results in a hyperparameter set, but not yet in the final model. The final model is obtained by training with this hyperparameter set and all data that was handed over to train().

    Regardless of the cross validation/bootstrap routine you chose, as long as the tuned hyperparameter set is the same, the final model will also be the same (at least for deterministic training routines such as lm()).

    So, even if you had tuned, still the same model would result: the one with intercept = TRUE.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.