2
$\begingroup$

The caret R package includes functions to compare models via their resampling distributions. Specifically, it prescribes fitting multiple models using the same resampling profiles (i.e. same versions of the training data) then conducting a paired t-test on the matched differences (see: http://topepo.github.io/caret/model-training-and-tuning.html#between).

Questions:

(a) Does the result of the paired t-test on the resampling distributions not simply reflect resampling variance (i.e. more repeats, lower p-value, see example code below) rather than performance differences between the models?

(b) Isn't there within-resample correlation (because training sets will overlap across folds/repeats) that will violate the assumption of independent observations necessary for the t-test?

(c) How to correctly test the null hypothesis of equal performance between (repeated cross-validated) models via their resampling distributions (using caret)?

require(caret)
data(iris)

set.seed(2)
p_values <- NA
repeats <- NA

for (i in 1:35){
train_control <- trainControl(
method="repeatedcv",
number=10,
repeats=i, 
returnResamp="final")

m1 <- train(Sepal.Length~Sepal.Width, data=iris, trControl=train_control, method="lm")
m2 <- train(Sepal.Length~Sepal.Width+ Petal.Length, data=iris, trControl=train_control, method="lm")

resamps <- resamples(list(m1 ,m2))

p_values[i]<- diff(resamps)$statistics$Rsquared$Model1.diff.Model2$p.value
repeats[i]<-i
}
plot(-log10(p_values),repeats)
$\endgroup$
  • $\begingroup$ I think this is an interesting question so +1, but I'd also be interested in why and how you think any within-resample-correlation would affect the (T-)test which is performed. Have you performed simulations which point in this direction? Or know of studies which results support this statement? Further, isn't a paired T-test performed specifically to account for some inherent correlation between measurements (hence the 'paired' adjective)? $\endgroup$ – IWS Apr 12 '17 at 12:36
  • $\begingroup$ Thanks for your response! The 'paired' in paired t-test refers to the fact that the tested differences between (!) models are matched (here this is achieved by holding resampling profiles constant for both models). However, the issue I am concerned about is correlation/non-independence between resamples for each model (bc cross-validation folds/repeats overlap), which might violate the assumption of independent observations necessary for the t-test. $\endgroup$ – user156902 Apr 12 '17 at 12:44
  • $\begingroup$ I agree, the models scores are correlated across resamples. So if one model scores high in a certain resample, the other model will probably do so to. However, to me, taking the difference between the two is the way of taking this correlation out of the equation. In this case the size of the difference will vary only because of the random sample drawn. If you'd then look at the distribution of differences found, or pool the differences found while taking into account the variance of the differences across resamples, I think it is a pretty reasonable (T-)test of your null hypothesis. $\endgroup$ – IWS Apr 12 '17 at 12:54
  • $\begingroup$ DO correct me if I'm wrong though... $\endgroup$ – IWS Apr 12 '17 at 12:55
  • $\begingroup$ Thanks for engaging with this! I am not worried about correlatedness BETWEEN models (i.e. in the above example, the rows on resamps$values are matched), but about correlatedness of resamples WITHIN a model, so dependency of the values within resamps$values$Model1~Rsquared. As far as I know the paired t-test is made for testing paired observations of two samples, but the observations within each sample (here model) need to be independent ('assumption of independent observations')? $\endgroup$ – user156902 Apr 12 '17 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.