3
$\begingroup$

I just need a simple yes/no answer (hopefully yes) to confirm I haven't done something stupid here - I'm doing some data analysis and looking at the correlation of 2 variables X and Y over the past 6 years. My correlation over these 6 years comes out in MS Excel as 96% (that is, the usual definition of correlation as detailed here http://office.microsoft.com/en-us/excel-help/correl-HP005209023.aspx): however, my correlations for each of these 6 years are 73%, 84%, 95%, 42%, 84% and 82% supposedly.

Is this possible, that the 6yr correlation is so much higher than any of the individual ones? I was surprised at how much higher the 6-year correlation was than any of the yearly ones. From drawing a picture this seems plausible, but I couldn't find any simple mathematical justification for the fact without things in the formula getting extremely messy, and there are a few hundred data points per year so it's not really feasible to check my data by hand.

Improve this question
$\endgroup$
1
  • $\begingroup$ It's also possible for the correlation over 6 years to have the opposite sign of the correlation in each of the years. See Simpson's paradox. en.wikipedia.org/wiki/Simpson's_paradox $\endgroup$
    – Douglas Zare
    Commented Nov 1, 2012 at 18:20

2 Answers 2

Reset to default
2
$\begingroup$

There's no reason this couldn't happen, that I can see.

Further, one year (if I am reading your question correctly) has a correlation of 95% and your overall is 96%, which isn't so different.

One way this could happen is if year has an effect on both x and y; in this case, you could have individual correlations of 0 and an overall one that is very high:

set.seed(1021827)
year <- rep(2000:2005, each = 100)
x <- year*3 + rnorm(600)
y <- year*3 + rnorm(600)
cor(x,y)
cor(x[year == 2000],y[year == 2000])
cor(x[year == 2001],y[year == 2001])
cor(x[year == 2002],y[year == 2002])
cor(x[year == 2003],y[year == 2003])
cor(x[year == 2004],y[year == 2004])
cor(x[year == 2005],y[year == 2005])

In this case the relationship between x and y is confounded by time. But that isn't necessary given the values you gave.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ You probably want library(plyr); d <- data.frame(x, y, year); ddply(d, .(year), summarize, r=cor(x, y)) :-) $\endgroup$
    – chl
    Commented Nov 1, 2012 at 12:23
  • $\begingroup$ @chl Yeah, I really should learn plyr, it simplifies things. But my code does the same thing, doesn't it? (Just more verbose). (I am still more of a SAS person than an R person). $\endgroup$
    – Peter Flom
    Commented Nov 1, 2012 at 12:30
  • $\begingroup$ Agreed that one year wasn't so different, I guess it was more the 42% year which surprised me when I broke things down annually - looking at it again though I think you're completely right, it's clear if you draw the picture corresponding to your example that you get exactly the result I was asking about. Thanks for your help! $\endgroup$
    – Ben
    Commented Nov 1, 2012 at 12:36
  • $\begingroup$ Indeed, Peter, we got the same results. I agree that plyr can be really useful sometimes, although I tend to stick to base R functions. $\endgroup$
    – chl
    Commented Nov 1, 2012 at 12:43
1
$\begingroup$

Yes, this is possible. Indeed, it's not just possible but a fairly common situation. For example I was looking recently at time series data (1987-2011) for world oil and natural gas prices (adjusted for general price inflation), and found that the correlation over the whole time period was higher than that over either half of the period. The reason is that both prices are on an upward trend, but the correlation coefficients for the shorter periods are more influenced by differences in fluctuations about the trend.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.