20
$\begingroup$

Let's say I test how variable Y depends on variable X under different experimental conditions and obtain the following graph:

enter image description here

The dash lines in the graph above represent linear regression for each data series (experimental setup) and the numbers in the legend denote the Pearson correlation of each data series.

I would like to calculate the "average correlation" (or "mean correlation") between X and Y. May I simply average the r values? What about the "average determination criterion", $R^2$? Should I calculate the average r and than take the square of that value or should I compute the average of individual $R^2$'s?

$\endgroup$
15
$\begingroup$

The simple way is to add a categorical variable $z$ to identify the different experimental conditions and include it in your model along with an "interaction" with $x$; that is, $y \sim z + x\#z$. This conducts all five regressions at once. Its $R^2$ is what you want.

To see why averaging individual $R$ values may be wrong, suppose the direction of the slope is reversed in some of the experimental conditions. You would average a bunch of 1's and -1's out to around 0, which wouldn't reflect the quality of any of the fits. To see why averaging $R^2$ (or any fixed transformation thereof) is not right, suppose that in most experimental conditions you had only two observations, so that their $R^2$ all equal $1$, but in one experiment you had a hundred observations with $R^2=0$. The average $R^2$ of almost 1 would not correctly reflect the situation.

$\endgroup$
  • 1
    $\begingroup$ pardon my ignorance, but what does the # sign in your answer mean? $\endgroup$ – Boris Gorelik Mar 10 '11 at 7:55
  • 1
    $\begingroup$ I think your answer is a very good one for the implied definition of correlation used. What if they meant it as mean standardized slope (perhaps implied by the figure)? In that case you do want negatives and positives to cancel. You're dead on about the sample size issue. Also, consider moving your comment into your answer. $\endgroup$ – John May 12 '13 at 16:26
  • $\begingroup$ Do you want the $R^2$ or the adjusted $R^2$? $\endgroup$ – russellpierce Jul 23 '13 at 10:20
  • $\begingroup$ @whuber in your initial comment there, your mean that the correlation could be $\pm 1$; the $R^2$ in each case is $1$. (I realize this is only a typing or editing issue; it doesn't change your point, but it may mislead.) $\endgroup$ – Glen_b Aug 19 '15 at 1:31
  • $\begingroup$ @rpierce In the second paragraph it makes no difference to the ideas if you use adjusted $R^2$--simply imagine sets of three, rather than two points, that are nearly collinear. Their adjusted $R^2$ can be arbitrarily close to $1$. $\endgroup$ – whuber Apr 27 '16 at 11:24
24
$\begingroup$

For Pearson correlation coefficients, it is generally appropriate to transform the r values using a Fisher z transformation. Then average the z-values and convert the average back to an r value.

I imagine it would be fine for a Spearman coefficient as well.

Here's a paper and the wikipedia entry.

$\endgroup$
  • 1
    $\begingroup$ +1; This answer seems more appropriate and general than the accepted answer however in the particular use case wouldn't it fall apart for r values of 1? Is something like an emperical logit reasonable here where one would just "add" a datapoint that lacks the correlation? If so, where would one add it? Would one have to conduct a monte carlo sim grabing two random variables from the source distributions? Alternatively would one just adjust r to some value slightly less than 1? By how far should one adjust? $\endgroup$ – russellpierce Jul 23 '13 at 10:30
3
$\begingroup$

The average correlation can be meaningul. Also consider the distribution of correlations (for example, plot a histogram).

But as I understand it, for each individual you have some ranking of $n$ items plus predicted rankings of those items for that individual, and you're looking at the correlation between an individual's rankings and the predicted ones.

In this case, it may be that the correlation is not the best measure of how well the algorithm is making predictions. For example, imagine that the algorithm gets the first 100 items perfectly and the next 200 items totally messed up, vs the opposite. It could be that you care only about the quality of the top rankings. In this case, you might look at the sum of the absolute differences between the individual's ranking and the predicted ranking, but only among the individual's top $m$ items.

$\endgroup$
1
$\begingroup$

What about using mean squared predicted eror (MSPE) for the performance of the algorithm? This is a standard approach to what you are trying to do, if you are trying to compare predictive performance among a set of algorithms.

$\endgroup$
  • $\begingroup$ I'm not sure why this post stats.stackexchange.com/questions/17129/… was merged with this one. They are actually asking two different questions in my opintion -- there are two different goals. $\endgroup$ – StatsStudent Dec 14 '11 at 18:13
  • 1
    $\begingroup$ You are correct: they are different questions. I have voted to re-open the other post (although what effect that might have is unclear). I apologize for not seeing your comment: if you had instead flagged that post it would have come to our attention several years sooner! $\endgroup$ – whuber Mar 3 '14 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.