# Perform analysis about model prediction diferrences using CARET

Using CARET package one can perform an analysis on differences between various models obtained using a dataset (the training dataset, trainSet) and the model that best fits the trainSet:

For example, two models were determined using train function, marsFit and rfFit:

resamps <- resamples(list(MARSCV = marsFit,RFCV = rfFit))
summary(resamps)
#Since models are fit on the same versions of the training data we can compute the differences, then use a simple t-test to evaluate the null hypothesis that there is no difference between models.
modelDifferences <- diff(resamps)
summary(modelDifferences)
bwplot(modelDifferences, layout = c(2, 1),
scales = list(x = list(relation="free")))


Is there a way to do a similar analysis (t-test to compare mean values for example) using the model predictions on the test set (separate dataset than trainSet)? CARET has a predict function from which one can evaluate the differences between the observed (testSet) and the predicted (model output) values.

Thanks!

if you want to look at "the differences between the observed (testSet) and the predicted (model output) values", then sure:

Here is an example:

library(caret)

## simulate some regression data
set.seed(1)
dat1 <- SLC14_1(500)
dat2 <- SLC14_1(500)

set.seed(2)
mod <- train(y ~ ., data = dat1,
method = "earth",
tuneLength = 5)

test_pred <- predict(mod, dat2)


then:

> postResample(test_pred, dat2$y) RMSE Rsquared 13.4965727 0.5770512 > t.test(test_pred - dat2$y)

One Sample t-test

data:  test_pred - dat2$y t = 0.7799, df = 499, p-value = 0.4358 alternative hypothesis: true mean is not equal to 0 95 percent confidence interval: -0.7154253 1.6572671 sample estimates: mean of x 0.4709209 > cor.test(test_pred, dat2$y)

Pearson's product-moment correlation

data:  test_pred and dat2$y t = 26.0662, df = 498, p-value < 2.2e-16 alternative hypothesis: true correlation is not equal to 0 95 percent confidence interval: 0.7199033 0.7944112 sample estimates: cor 0.7596388  resamples is really about comparing multiple models though (as opposed to a single model vs the true values. In that case, you could compare residuals: set.seed(2) mod2 <- train(y ~ ., data = dat1, method = "lm") test_pred2 <- predict(mod2, dat2)  and > t.test(test_pred - dat2$y, test_pred2 - dat2$y, paired = TRUE) Paired t-test data: test_pred - dat2$y and test_pred2 - dat2\$y
t = -2.3911, df = 499, p-value = 0.01717
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-2.9113903 -0.2850041
sample estimates:
mean of the differences
-1.598197


Max

• Hi Max! Not wanting to abuse your help, does it matter the column order of the data matrix in predict(model, data) if the columns have names which are also used in the training of model? Commented Sep 2, 2014 at 2:46
• That really depends on the underlying model R code and whether the formula or non-formula interface is used there, so it is hard to say Commented Sep 3, 2014 at 19:10