# Correlations over 6yrs higher than any individual 1yr correlation over the same period

I just need a simple yes/no answer (hopefully yes) to confirm I haven't done something stupid here - I'm doing some data analysis and looking at the correlation of 2 variables X and Y over the past 6 years. My correlation over these 6 years comes out in MS Excel as 96% (that is, the usual definition of correlation as detailed here http://office.microsoft.com/en-us/excel-help/correl-HP005209023.aspx): however, my correlations for each of these 6 years are 73%, 84%, 95%, 42%, 84% and 82% supposedly.

Is this possible, that the 6yr correlation is so much higher than any of the individual ones? I was surprised at how much higher the 6-year correlation was than any of the yearly ones. From drawing a picture this seems plausible, but I couldn't find any simple mathematical justification for the fact without things in the formula getting extremely messy, and there are a few hundred data points per year so it's not really feasible to check my data by hand.

-
It's also possible for the correlation over 6 years to have the opposite sign of the correlation in each of the years. See Simpson's paradox. en.wikipedia.org/wiki/Simpson's_paradox –  Douglas Zare Nov 1 '12 at 18:20

There's no reason this couldn't happen, that I can see.

Further, one year (if I am reading your question correctly) has a correlation of 95% and your overall is 96%, which isn't so different.

One way this could happen is if year has an effect on both x and y; in this case, you could have individual correlations of 0 and an overall one that is very high:

set.seed(1021827)
year <- rep(2000:2005, each = 100)
x <- year*3 + rnorm(600)
y <- year*3 + rnorm(600)
cor(x,y)
cor(x[year == 2000],y[year == 2000])
cor(x[year == 2001],y[year == 2001])
cor(x[year == 2002],y[year == 2002])
cor(x[year == 2003],y[year == 2003])
cor(x[year == 2004],y[year == 2004])
cor(x[year == 2005],y[year == 2005])


In this case the relationship between x and y is confounded by time. But that isn't necessary given the values you gave.

-
You probably want library(plyr); d <- data.frame(x, y, year); ddply(d, .(year), summarize, r=cor(x, y)) :-) –  chl Nov 1 '12 at 12:23
@chl Yeah, I really should learn plyr, it simplifies things. But my code does the same thing, doesn't it? (Just more verbose). (I am still more of a SAS person than an R person). –  Peter Flom Nov 1 '12 at 12:30
Agreed that one year wasn't so different, I guess it was more the 42% year which surprised me when I broke things down annually - looking at it again though I think you're completely right, it's clear if you draw the picture corresponding to your example that you get exactly the result I was asking about. Thanks for your help! –  Ben Nov 1 '12 at 12:36
Indeed, Peter, we got the same results. I agree that plyr can be really useful sometimes, although I tend to stick to base R functions. –  chl Nov 1 '12 at 12:43