# Mean when computing correlation between samples of unequal size

This question is in some way similar to this one, but about another nuance. I have two time series (update: stationary - with both mean and variance equal over time) with missing values in one of them, such as

• 1,2,3
• 1,Absent,3

(in reality I have many more observations). I want to compute their correlation, so I will not use the 2nd time-point in these data. But should I use it for computing the mean of the first series?

I think that including this point in computing the mean will lead to a more precise estimate of the population mean for the first variable, but don't know whether it is correct though.

-
What is a "suck point"? A time with missing data? –  whuber Oct 31 '11 at 18:52
Yes, sorry. I'm not very good in English. –  Ivan Sopov Oct 31 '11 at 18:57
That's ok. Could you clarify your intentions, though? I am wondering whether you are talking about computing the mean in the first time series for the purpose of developing a variance estimate (which will enter into the correlation calculation) or for some separate purpose. –  whuber Oct 31 '11 at 19:12
upload.wikimedia.org/wikipedia/en/math/d/1/f/… - both for covariance and for variances of series. My intent was to compute mean for first series only once and use it every time when it is needed in computation of correlation. However I need not save anything except Pearson correlation coefficient. –  Ivan Sopov Oct 31 '11 at 19:55
As @PeterFlom mentions below, using Pearson's $r_{xy}$ is probably not a good idea if you have true time series data. Could you provide more information about your data so someone can provide a more informed answer? –  Jason Morgan Nov 1 '11 at 0:11
show 1 more comment

Let's call the first series $X_t$ and the second series $Y_t$ and denote the length of each series with $T$. You are asking if using all of the information from $X_t$ to calculate its mean in the computation of the correlation will make the estimate of the correlation more efficient. But you have to be precise about the exact way you want to calculate that correlation. Remember that:

$Cor(X_t, Y_t) = \frac{Cov(X_t, Y_t)}{\sqrt{Var(X_t) Var(Y_t)}}$,

where

$Cov(X_t, Y_t) = \frac{\sum (X_t - \bar{X}_t)(Y_t - \bar{Y}_t)}{T - 1}$,

$Var(X_t) = \frac{\sum (X_t - \bar{X}_t)^2}{T-1}$,

and

$Var(Y_t) = \frac{\sum (Y_t - \bar{Y}_t)^2}{T-1}$.

Now suppose half of the $Y_t$ values are missing. Then $Cov(X_t, Y_t)$ is calculated based on $T/2$ observations (even when $\bar{X}_t$ is calculated from all $T$ observations), so one would also divide by $T/2 - 1$ and not $T-1$. For $Var(X_t)$, one could use all $T$ observations (not only for calculating $\bar{X}_t$, but also for calculating the entire term, so one would still divide by $T-1$ here). And for $Var(Y_t)$, one only has $T/2$ observations, so one would divide by $T/2 - 1$.

I suppose this is what you mean. And you are not sure whether this estimator is more efficient than just doing pairwise deletion. If in doubt, you can always examine a question like this with a simulation study. Below is a simple example using R to simulate two time series, $X_t$ and $Y_t$, that are stationary and that are uncorrelated. I first calculate the observed correlation based on the full data. Then I induce random missingness on 50% of the $Y_t$ values. Then I calculate the correlation only based on the pairs of data points where $Y_t$ is not missing (i.e., pairwise deletion). And then I calculate the correlation as described above. I repeat this process many times and then calculate the mean correlation for each method (the three means should ideally be zero, indicating no bias in the estimators) and the standard deviation (indicating the efficiency of the estimators; smaller is obviously better).

iters <- 100000      ### number of iterations
nt <- 50             ### length of the series

ri <- matrix(NA, nrow=iters, ncol=3)
miss <- c(rep(1, nt/2), rep(0, nt/2))

for (i in 1:iters) {

xt <- 0 + 0*(1:nt) + rnorm(nt)
yt <- 0 + 0*(1:nt) + rnorm(nt)

ri[i,1] <- cor(xt, yt)

yt[sample(miss) == 1] <- NA
not.miss <- !is.na(yt)

ri[i,2] <- cor(xt[not.miss], yt[not.miss])

mxt  <- mean(xt)
myt  <- mean(yt[not.miss])
num <- sum((xt[not.miss] - mxt) * (yt[not.miss] - myt)) / (sum(not.miss) - 1)
den <- sqrt(sum((xt - mxt)^2)/(nt-1) * sum((yt[not.miss] - myt)^2)/(sum(not.miss)-1))

ri[i,3] <-  num / den

}

round(apply(ri, 2, mean), 2)
round(apply(ri, 2, sd), 2)


If I run this, I get:

> round(apply(ri, 2, mean), 2)
[1] 0 0 0
> round(apply(ri, 2, sd), 2)
[1] 0.14 0.20 0.20


indicating no bias in the three estimators. The method that uses pairwise deletion is as efficient as the method that uses all of the information from $X_t$. So, apparently, you do not gain any efficiency in calculating the correlation by doing that.

Suppose there is actually a trend in time in both series. This will in fact induce a correlation between the two series. You can simulate this by replacing the respective lines with:

xt <- 0 + 0.15*(1:nt) + rnorm(nt)
yt <- 0 + 0.15*(1:nt) + rnorm(nt)


Rerunning the simulation yields:

> round(apply(ri, 2, mean), 2)
[1] 0.83 0.83 0.83
> round(apply(ri, 2, sd), 2)
[1] 0.03 0.05 0.10


so all three estimators give you the same answer (on average). But pairwise deletion is actually now more efficient than using all available information from $X_t$.

Edit: You can also induce a correlation between $X_t$ and $Y_t$ without a trend, for example, by replacing the respective lines with:

xt <- 0 + 0*(1:nt) + rnorm(nt)
yt <- 0 + 0*(1:nt) + 1.0*xt + rnorm(nt)


Rerunning the simulation yields:

> round(apply(ri, 2, mean), 2)
[1] 0.7 0.7 0.7
> round(apply(ri, 2, sd), 2)
[1] 0.07 0.11 0.15


and again, pairwise deletion is more efficient.

So, I would say: No, don't do this.

-
I really like the idea of running a simulation. It has also been talked about in this forum about generating random numbers with a predefined correlation. It would be cool to do that, and then randomly select missing numbers and compare. (Like I think you did). (I think you just used a correlation of zero, correct?) –  Adam Nov 2 '11 at 5:41
I simulated the data under two conditions: first when the true correlation is zero and then when the true correlation is non-zero. –  Wolfgang Nov 2 '11 at 5:54
Sorry, you wrote " This will in fact induce a correlation between the two series. You can simulate this by replacing the respective lines with:"..I missed that somehow, thanks. –  Adam Nov 2 '11 at 6:03
Oh, I'm bad at R - I understand it normally, but cannot write anything myself (previously I ported some code from R to Java). How can I modify it to test correlated series without trend? - Generate them as random int 1-10 (common for two series) + random value 0.0-1.0 (different for two series). (In Java this will take some minutes to write, I presume that it is easy modification to this code) –  Ivan Sopov Nov 2 '11 at 9:18
I updated my answer to show how you can do this. –  Wolfgang Nov 2 '11 at 16:46
@Peter I see your point that if series $X_t$ and $Y_t$ have some increasing or decreasing trend as a function of time, then this will induce a correlation between $X_t$ and $Y_t$. But I still don't see why that implies that correlating two time series is a bad idea, if what I am interested in is the correlation between the two series. Nothing about that correlation implies anything about causation. Could you elaborate why this is generally a bad idea? –  Wolfgang Nov 2 '11 at 1:00