I have two models, A and B. I have performed 10-fold cross-validation on both of them, so now I have 10 $R^2$ scores for each.

How can I determine whether one is significantly better than the other? I fear that calling $A$ the winner iff $$\text{mean}(A)-\text{SEM}(A) > \text{mean}(B)+\text{SEM}(B)$$ is perhaps not the correct way to do it.


1 Answer 1


Well, assuming all of your modelling assumptions are fulfilled, the modelling techniques do not differ and also assuming $R^2$ is the only model characteristic that differs between the models, I would consider a Student’s t-test (assumptions permitting) or Wilcoxon-Mann-Whitney test on the $R^2$s. In any case I would try to get a larger sample (costs permitting).

You can read more about different model selection measures, particularly AIC, here.

If different modelling techniques were used you may find this interesting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.