How to show whether the average coefficients of determination from one regression technique are better than another across many objects?

Question

I have 50,000 objects on which I have performed two different types of regression. Using cross validation, I obtained the average $R^2$ score from each model on each of the objects. So now I have a list of 50k pairs of average $R^2$ scores.

I have used a paired Wilcoxon-Mann-Whitney U test and found that their means differ significantly:

> wilcox.test(first$R.2, second$R.2, paired=TRUE)

    Wilcoxon signed rank test with continuity correction

data:  first$R.2 and second$R.2
V = 34503366, p-value < 2.2e-16
alternative hypothesis: true location shift is not equal to 0

Is this the right approach? I figure this is the right test because $R^2$ is bounded above by 1 but unbounded below so I can't really make any assumption about the distribution.

How I can report this in my paper? Do I just say that I used the Wilcoxon test and obtained a p-value less than 2.2e-16?

I also wanted to show the mean and median $R^2$s from each of the techniques. But should I just show point estimates of these values, or should I also show SDs and MADs or SEMs or what? Should I bootstrap confidence intervals for them and see whether they overlap?

Answer 1

Most important thing is to make sure both models initially are significant and giving accurate depiction of underlying data. Another layer to add to your process is the ROC curve which allows you to compare false positive performance (Type I/II errors). really powerful and there are some packages out there that make it go smoothly.

The link below contains link to R package that deals with it.

http://cran.r-project.org/web/packages/pROC/pROC.pdf