Assuming I have a training set that is used to train 3 different random forest models. The values of the response variable $y$ are the same for all of the 3 models. But the predictors $x$ differ between the 3 models, although some of the predictors $x$ are used in every model. So the predictors $x$ of the 3 training sets are not the same. Thus, 3 different models are trained using the same $y$ but different $x$.

A error measure is generated for every model using k-fold cross-validation. The test sets differ in $x$, the ground truth of $y$ is the same. The resulting predictions $\hat y$ should be the same if the models perform equally and thus the error measure would be also the same for each model.

Assume the error measures of the different models have equal variances and are normally distributed. Could I perform a one-way within-subjects ANOVA (ANOVA for correlated samples)? Or is the independence assumption violated, because I am using the same training sets (that differ only in $x$ and not in $y$ between the models) to build the 3 models and the same test sets (that also differ only in $x$ and not in $y$ between the models)?

Does the independence assumption refer to independet data within each group (model)? For the training data it would then be violated, because they overlap due to k-fold cross-validation. Only the test data don't overlap and would thus be independent.

Should I rather use a non-parametric hypothesis test as the Friedman test?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.