1
$\begingroup$

Using CARET package one can perform an analysis on differences between various models obtained using a dataset (the training dataset, trainSet) and the model that best fits the trainSet:

For example, two models were determined using train function, marsFit and rfFit:

resamps <- resamples(list(MARSCV = marsFit,RFCV = rfFit))
summary(resamps)
#Since models are fit on the same versions of the training data we can compute the differences, then use a simple t-test to evaluate the null hypothesis that there is no difference between models.
modelDifferences <- diff(resamps)
summary(modelDifferences)
bwplot(modelDifferences, layout = c(2, 1),
       scales = list(x = list(relation="free")))

Is there a way to do a similar analysis (t-test to compare mean values for example) using the model predictions on the test set (separate dataset than trainSet)? CARET has a predict function from which one can evaluate the differences between the observed (testSet) and the predicted (model output) values.

Thanks!

$\endgroup$

1 Answer 1

2
$\begingroup$

if you want to look at "the differences between the observed (testSet) and the predicted (model output) values", then sure:

Here is an example:

library(caret)

## simulate some regression data
set.seed(1)
dat1 <- SLC14_1(500)
dat2 <- SLC14_1(500)

set.seed(2)
mod <- train(y ~ ., data = dat1, 
             method = "earth",
             tuneLength = 5)

test_pred <- predict(mod, dat2)

then:

> postResample(test_pred, dat2$y)
          RMSE   Rsquared 
    13.4965727  0.5770512 
    > t.test(test_pred - dat2$y)

    One Sample t-test

data:  test_pred - dat2$y
t = 0.7799, df = 499, p-value = 0.4358
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -0.7154253  1.6572671
sample estimates:
mean of x 
0.4709209 

> cor.test(test_pred, dat2$y)

    Pearson's product-moment correlation

data:  test_pred and dat2$y
t = 26.0662, df = 498, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.7199033 0.7944112
sample estimates:
      cor 
0.7596388 

resamples is really about comparing multiple models though (as opposed to a single model vs the true values. In that case, you could compare residuals:

set.seed(2)
mod2 <- train(y ~ ., data = dat1, 
              method = "lm")    
test_pred2 <- predict(mod2, dat2)

and

> t.test(test_pred - dat2$y, test_pred2 - dat2$y, paired = TRUE)

    Paired t-test

data:  test_pred - dat2$y and test_pred2 - dat2$y
t = -2.3911, df = 499, p-value = 0.01717
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.9113903 -0.2850041
sample estimates:
mean of the differences 
              -1.598197 

Max

$\endgroup$
2
  • $\begingroup$ Hi Max! Not wanting to abuse your help, does it matter the column order of the data matrix in predict(model, data) if the columns have names which are also used in the training of model? $\endgroup$
    – jpcgandre
    Sep 2, 2014 at 2:46
  • 1
    $\begingroup$ That really depends on the underlying model R code and whether the formula or non-formula interface is used there, so it is hard to say $\endgroup$
    – topepo
    Sep 3, 2014 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.