# What are RMSE SD and Rsquared SD metrics in resampling results using R package:caret?

I've been doing predictive modelling with R package caret. When resampling regression models, I get the traditional RMSE and Rsquared metrics, but also RMSE SD and Rsquared SD, for which I haven't found explanation in the manuals or documentations. Please, could anyone enlighten me?

Reproducible code:

library(mlbench)
data(BostonHousing)
library(caret)

ctrl <- trainControl(method = "cv", number = 2)
lmFit <- train(medv ~ ., data = BostonHousing, method = "lm", trControl = ctrl)
print(lmFit)


outputs:

Linear Regression

506 samples
13 predictors

No pre-processing
Resampling: Cross-Validated (2 fold)

Summary of sample sizes: 254, 252

Resampling results

RMSE  Rsquared  RMSE SD  Rsquared SD
4.91  0.721     0.202    0.00304



Cheers

Maria

It is the standard deviation of the resamples:

> lmFit$resample RMSE Rsquared Resample 1 4.702857 0.7283872 Fold1 2 5.266187 0.6838433 Fold2 > apply(lmFit$resample[, 1:2], 2, sd)
RMSE   Rsquared
0.39833479 0.03149727
> lmFit
Linear Regression

506 samples
13 predictors

No pre-processing
Resampling: Cross-Validated (2 fold)

Summary of sample sizes: 253, 253

Resampling results

RMSE  Rsquared  RMSE SD  Rsquared SD
4.98  0.706     0.398    0.0315


Max

If you use crossvalidation, it is very convenient to retrieve the metrics through the caret model objects. This is especially useful if you train several models as you can retrieve the MAE, RMSE and Rsquared all at once.

Let's say you have trained two models, lm and gbm, with 10-fold cross-validation by train(preProcess(method = "cv", number = 10).

Then you can retrieve the metrics by

models.list <- list(fit.lm, fit.gbm)

models.list %>%
# get resamples results
resamples %>% .\$values %>%
# only select numeric columns
select_if(is.numeric) %>%
# calculate standard deviation
summarise_all(.funs = sd)


This returns the standard deviations for all metrics:

      lm~MAE    lm~RMSE lm~Rsquared    gbm~MAE   gbm~RMSE gbm~Rsquared
1 0.02168322 0.02877163  0.03492542 0.02834266 0.03550696   0.04176382


Of course, this is even more useful if you have more trained models in your model list.