1
$\begingroup$

I have 50,000 objects on which I have performed two different types of regression. Using cross validation, I obtained the average $R^2$ score from each model on each of the objects. So now I have a list of 50k pairs of average $R^2$ scores.

I have used a paired Wilcoxon-Mann-Whitney U test and found that their means differ significantly:

> wilcox.test(first$R.2, second$R.2, paired=TRUE)

    Wilcoxon signed rank test with continuity correction

data:  first$R.2 and second$R.2
V = 34503366, p-value < 2.2e-16
alternative hypothesis: true location shift is not equal to 0

Is this the right approach? I figure this is the right test because $R^2$ is bounded above by 1 but unbounded below so I can't really make any assumption about the distribution.

How I can report this in my paper? Do I just say that I used the Wilcoxon test and obtained a p-value less than 2.2e-16?

I also wanted to show the mean and median $R^2$s from each of the techniques. But should I just show point estimates of these values, or should I also show SDs and MADs or SEMs or what? Should I bootstrap confidence intervals for them and see whether they overlap?

$\endgroup$
  • $\begingroup$ $R^2$ is bounded from below by 0. $\endgroup$ – James Aug 28 '14 at 18:11
  • 2
    $\begingroup$ No it's not stats.stackexchange.com/a/12991/2488 $\endgroup$ – rhombidodecahedron Aug 28 '14 at 18:35
  • $\begingroup$ Why didn't you include the intercept? $\endgroup$ – James Aug 28 '14 at 20:08
  • $\begingroup$ I do have the intercept. That is not the only circumstance in which the $R^2$ can be negative. $\endgroup$ – rhombidodecahedron Aug 28 '14 at 20:34
0
$\begingroup$

Most important thing is to make sure both models initially are significant and giving accurate depiction of underlying data. Another layer to add to your process is the ROC curve which allows you to compare false positive performance (Type I/II errors). really powerful and there are some packages out there that make it go smoothly.

The link below contains link to R package that deals with it.

http://cran.r-project.org/web/packages/pROC/pROC.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.