I have two models, A and B. I have performed 10-fold cross-validation on both of them, so now I have 10 $R^2$ scores for each.

How can I determine whether one is significantly better than the other? I fear that calling $A$ the winner iff $$\text{mean}(A)-\text{SEM}(A) > \text{mean}(B)+\text{SEM}(B)$$ is perhaps not the correct way to do it.


Well, assuming all of your modelling assumptions are fulfilled, the modelling techniques do not differ and also assuming $R^2$ is the only model characteristic that differs between the models, I would consider a Student’s t-test (assumptions permitting) or Wilcoxon-Mann-Whitney test on the $R^2$s. In any case I would try to get a larger sample (costs permitting).

You can read more about different model selection measures, particularly AIC, here.

If different modelling techniques were used you may find this interesting.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.