What are RMSE SD and Rsquared SD metrics in resampling results using R package:caret?

Question

I've been doing predictive modelling with R package caret. When resampling regression models, I get the traditional RMSE and Rsquared metrics, but also RMSE SD and Rsquared SD, for which I haven't found explanation in the manuals or documentations. Please, could anyone enlighten me?

Reproducible code:

library(mlbench)
data(BostonHousing)
library(caret)

ctrl <- trainControl(method = "cv", number = 2)
lmFit <- train(medv ~ ., data = BostonHousing, method = "lm", trControl = ctrl)
print(lmFit)

outputs:

Linear Regression 

506 samples
13 predictors

No pre-processing
Resampling: Cross-Validated (2 fold) 

Summary of sample sizes: 254, 252 

Resampling results

RMSE  Rsquared  RMSE SD  Rsquared SD
4.91  0.721     0.202    0.00304    

Cheers

Maria

Answer 1

It is the standard deviation of the resamples:

> lmFit$resample
          RMSE  Rsquared Resample
    1 4.702857 0.7283872    Fold1
    2 5.266187 0.6838433    Fold2
> apply(lmFit$resample[, 1:2], 2, sd)
      RMSE   Rsquared 
0.39833479 0.03149727 
> lmFit
Linear Regression 

506 samples
 13 predictors

No pre-processing
Resampling: Cross-Validated (2 fold) 

Summary of sample sizes: 253, 253 

Resampling results

  RMSE  Rsquared  RMSE SD  Rsquared SD
  4.98  0.706     0.398    0.0315   

Max

Answer 2

If you use crossvalidation, it is very convenient to retrieve the metrics through the caret model objects. This is especially useful if you train several models as you can retrieve the MAE, RMSE and Rsquared all at once.

Let's say you have trained two models, lm and gbm, with 10-fold cross-validation by train(preProcess(method = "cv", number = 10).

Then you can retrieve the metrics by

models.list <- list(fit.lm, fit.gbm)

models.list %>% 
  # get resamples results
  resamples %>% .$values %>% 
  # only select numeric columns
  select_if(is.numeric) %>% 
  # calculate standard deviation
  summarise_all(.funs = sd) 

This returns the standard deviations for all metrics:

      lm~MAE    lm~RMSE lm~Rsquared    gbm~MAE   gbm~RMSE gbm~Rsquared
1 0.02168322 0.02877163  0.03492542 0.02834266 0.03550696   0.04176382 

Of course, this is even more useful if you have more trained models in your model list.