Perform analysis about model prediction diferrences using CARET

Question

Using CARET package one can perform an analysis on differences between various models obtained using a dataset (the training dataset, trainSet) and the model that best fits the trainSet:

For example, two models were determined using train function, marsFit and rfFit:

resamps <- resamples(list(MARSCV = marsFit,RFCV = rfFit))
summary(resamps)
#Since models are fit on the same versions of the training data we can compute the differences, then use a simple t-test to evaluate the null hypothesis that there is no difference between models.
modelDifferences <- diff(resamps)
summary(modelDifferences)
bwplot(modelDifferences, layout = c(2, 1),
       scales = list(x = list(relation="free")))

Is there a way to do a similar analysis (t-test to compare mean values for example) using the model predictions on the test set (separate dataset than trainSet)? CARET has a predict function from which one can evaluate the differences between the observed (testSet) and the predicted (model output) values.

Thanks!

Answer 1

if you want to look at "the differences between the observed (testSet) and the predicted (model output) values", then sure:

Here is an example:

library(caret)

## simulate some regression data
set.seed(1)
dat1 <- SLC14_1(500)
dat2 <- SLC14_1(500)

set.seed(2)
mod <- train(y ~ ., data = dat1, 
             method = "earth",
             tuneLength = 5)

test_pred <- predict(mod, dat2)

then:

> postResample(test_pred, dat2$y)
          RMSE   Rsquared 
    13.4965727  0.5770512 
    > t.test(test_pred - dat2$y)

    One Sample t-test

data:  test_pred - dat2$y
t = 0.7799, df = 499, p-value = 0.4358
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -0.7154253  1.6572671
sample estimates:
mean of x 
0.4709209 

> cor.test(test_pred, dat2$y)

    Pearson's product-moment correlation

data:  test_pred and dat2$y
t = 26.0662, df = 498, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.7199033 0.7944112
sample estimates:
      cor 
0.7596388 

resamples is really about comparing multiple models though (as opposed to a single model vs the true values. In that case, you could compare residuals:

set.seed(2)
mod2 <- train(y ~ ., data = dat1, 
              method = "lm")    
test_pred2 <- predict(mod2, dat2)

and

> t.test(test_pred - dat2$y, test_pred2 - dat2$y, paired = TRUE)

    Paired t-test

data:  test_pred - dat2$y and test_pred2 - dat2$y
t = -2.3911, df = 499, p-value = 0.01717
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.9113903 -0.2850041
sample estimates:
mean of the differences 
              -1.598197 

Max