26
$\begingroup$

Let's say I test how variable Y depends on variable X under different experimental conditions and obtain the following graph:

enter image description here

The dash lines in the graph above represent linear regression for each data series (experimental setup) and the numbers in the legend denote the Pearson correlation of each data series.

I would like to calculate the "average correlation" (or "mean correlation") between X and Y. May I simply average the r values? What about the "average determination criterion", $R^2$? Should I calculate the average r and than take the square of that value or should I compute the average of individual $R^2$'s?

Improve this question
$\endgroup$
0

4 Answers 4

Reset to default
17
$\begingroup$

The simple way is to add a categorical variable $z$ to identify the different experimental conditions and include it in your model along with an "interaction" with $x$; that is, $y \sim z + x\#z$. This conducts all five regressions at once. Its $R^2$ is what you want.

To see why averaging individual $R$ values may be wrong, suppose the direction of the slope is reversed in some of the experimental conditions. You would average a bunch of 1's and -1's out to around 0, which wouldn't reflect the quality of any of the fits. To see why averaging $R^2$ (or any fixed transformation thereof) is not right, suppose that in most experimental conditions you had only two observations, so that their $R^2$ all equal $1$, but in one experiment you had a hundred observations with $R^2=0$. The average $R^2$ of almost 1 would not correctly reflect the situation.

Improve this answer
$\endgroup$
6
  • 5
    $\begingroup$ pardon my ignorance, but what does the # sign in your answer mean? $\endgroup$
    – Boris Gorelik
    Commented Mar 10, 2011 at 7:55
  • 1
    $\begingroup$ I think your answer is a very good one for the implied definition of correlation used. What if they meant it as mean standardized slope (perhaps implied by the figure)? In that case you do want negatives and positives to cancel. You're dead on about the sample size issue. Also, consider moving your comment into your answer. $\endgroup$
    – John
    Commented May 12, 2013 at 16:26
  • $\begingroup$ Do you want the $R^2$ or the adjusted $R^2$? $\endgroup$
    – russellpierce
    Commented Jul 23, 2013 at 10:20
  • $\begingroup$ @whuber in your initial comment there, your mean that the correlation could be $\pm 1$; the $R^2$ in each case is $1$. (I realize this is only a typing or editing issue; it doesn't change your point, but it may mislead.) $\endgroup$
    – Glen_b
    Commented Aug 19, 2015 at 1:31
  • 1
    $\begingroup$ Could anyone tell me how to do that in Excel or Matlab ? $\endgroup$
    – kgk
    Commented Jul 26, 2017 at 6:29
33
$\begingroup$

For Pearson correlation coefficients, it is generally appropriate to transform the r values using a Fisher z transformation. Then average the z-values and convert the average back to an r value.

I imagine it would be fine for a Spearman coefficient as well.

Here's a paper and the wikipedia entry.

Improve this answer
$\endgroup$
1
  • $\begingroup$ What would be the analogous transformations for Cramer's V, Theil's U, or correlation ratios? $\endgroup$
    – michen00
    Commented Nov 1, 2021 at 3:26
3
$\begingroup$

The average correlation can be meaningul. Also consider the distribution of correlations (for example, plot a histogram).

But as I understand it, for each individual you have some ranking of $n$ items plus predicted rankings of those items for that individual, and you're looking at the correlation between an individual's rankings and the predicted ones.

In this case, it may be that the correlation is not the best measure of how well the algorithm is making predictions. For example, imagine that the algorithm gets the first 100 items perfectly and the next 200 items totally messed up, vs the opposite. It could be that you care only about the quality of the top rankings. In this case, you might look at the sum of the absolute differences between the individual's ranking and the predicted ranking, but only among the individual's top $m$ items.

Improve this answer
$\endgroup$
1
$\begingroup$

What about using mean squared predicted eror (MSPE) for the performance of the algorithm? This is a standard approach to what you are trying to do, if you are trying to compare predictive performance among a set of algorithms.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'm not sure why this post stats.stackexchange.com/questions/17129/… was merged with this one. They are actually asking two different questions in my opintion -- there are two different goals. $\endgroup$
    – StatsStudent
    Commented Dec 14, 2011 at 18:13
  • 1
    $\begingroup$ You are correct: they are different questions. I have voted to re-open the other post (although what effect that might have is unclear). I apologize for not seeing your comment: if you had instead flagged that post it would have come to our attention several years sooner! $\endgroup$
    – whuber ♦
    Commented Mar 3, 2014 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.