4
$\begingroup$

This question is in some way similar to this one, but about another nuance. I have two time series (update: stationary - with both mean and variance equal over time) with missing values in one of them, such as

  • 1,2,3
  • 1,Absent,3

(in reality I have many more observations). I want to compute their correlation, so I will not use the 2nd time-point in these data. But should I use it for computing the mean of the first series?

I think that including this point in computing the mean will lead to a more precise estimate of the population mean for the first variable, but don't know whether it is correct though.

Improve this question
$\endgroup$
6
  • $\begingroup$ What is a "suck point"? A time with missing data? $\endgroup$
    – whuber ♦
    Oct 31, 2011 at 18:52
  • $\begingroup$ Yes, sorry. I'm not very good in English. $\endgroup$
    – Ivan Sopov
    Oct 31, 2011 at 18:57
  • $\begingroup$ That's ok. Could you clarify your intentions, though? I am wondering whether you are talking about computing the mean in the first time series for the purpose of developing a variance estimate (which will enter into the correlation calculation) or for some separate purpose. $\endgroup$
    – whuber ♦
    Oct 31, 2011 at 19:12
  • 1
    $\begingroup$ upload.wikimedia.org/wikipedia/en/math/d/1/f/… - both for covariance and for variances of series. My intent was to compute mean for first series only once and use it every time when it is needed in computation of correlation. However I need not save anything except Pearson correlation coefficient. $\endgroup$
    – Ivan Sopov
    Oct 31, 2011 at 19:55
  • $\begingroup$ As @PeterFlom mentions below, using Pearson's $r_{xy}$ is probably not a good idea if you have true time series data. Could you provide more information about your data so someone can provide a more informed answer? $\endgroup$
    – Jason Morgan
    Nov 1, 2011 at 0:11

2 Answers 2

Reset to default
2
+100
$\begingroup$

Let's call the first series $X_t$ and the second series $Y_t$ and denote the length of each series with $T$. You are asking if using all of the information from $X_t$ to calculate its mean in the computation of the correlation will make the estimate of the correlation more efficient. But you have to be precise about the exact way you want to calculate that correlation. Remember that:

$Cor(X_t, Y_t) = \frac{Cov(X_t, Y_t)}{\sqrt{Var(X_t) Var(Y_t)}}$,

where

$Cov(X_t, Y_t) = \frac{\sum (X_t - \bar{X}_t)(Y_t - \bar{Y}_t)}{T - 1}$,

$Var(X_t) = \frac{\sum (X_t - \bar{X}_t)^2}{T-1}$,

and

$Var(Y_t) = \frac{\sum (Y_t - \bar{Y}_t)^2}{T-1}$.

Now suppose half of the $Y_t$ values are missing. Then $Cov(X_t, Y_t)$ is calculated based on $T/2$ observations (even when $\bar{X}_t$ is calculated from all $T$ observations), so one would also divide by $T/2 - 1$ and not $T-1$. For $Var(X_t)$, one could use all $T$ observations (not only for calculating $\bar{X}_t$, but also for calculating the entire term, so one would still divide by $T-1$ here). And for $Var(Y_t)$, one only has $T/2$ observations, so one would divide by $T/2 - 1$.

I suppose this is what you mean. And you are not sure whether this estimator is more efficient than just doing pairwise deletion. If in doubt, you can always examine a question like this with a simulation study. Below is a simple example using R to simulate two time series, $X_t$ and $Y_t$, that are stationary and that are uncorrelated. I first calculate the observed correlation based on the full data. Then I induce random missingness on 50% of the $Y_t$ values. Then I calculate the correlation only based on the pairs of data points where $Y_t$ is not missing (i.e., pairwise deletion). And then I calculate the correlation as described above. I repeat this process many times and then calculate the mean correlation for each method (the three means should ideally be zero, indicating no bias in the estimators) and the standard deviation (indicating the efficiency of the estimators; smaller is obviously better).

iters <- 100000      ### number of iterations
nt <- 50             ### length of the series

ri <- matrix(NA, nrow=iters, ncol=3)
miss <- c(rep(1, nt/2), rep(0, nt/2))

for (i in 1:iters) {

    xt <- 0 + 0*(1:nt) + rnorm(nt)
    yt <- 0 + 0*(1:nt) + rnorm(nt)

    ri[i,1] <- cor(xt, yt)

    yt[sample(miss) == 1] <- NA
    not.miss <- !is.na(yt)

    ri[i,2] <- cor(xt[not.miss], yt[not.miss])

    mxt  <- mean(xt)
    myt  <- mean(yt[not.miss])
    num <- sum((xt[not.miss] - mxt) * (yt[not.miss] - myt)) / (sum(not.miss) - 1)
    den <- sqrt(sum((xt - mxt)^2)/(nt-1) * sum((yt[not.miss] - myt)^2)/(sum(not.miss)-1))

    ri[i,3] <-  num / den

}

round(apply(ri, 2, mean), 2)
round(apply(ri, 2, sd), 2)

If I run this, I get:

> round(apply(ri, 2, mean), 2)
[1] 0 0 0
> round(apply(ri, 2, sd), 2)
[1] 0.14 0.20 0.20

indicating no bias in the three estimators. The method that uses pairwise deletion is as efficient as the method that uses all of the information from $X_t$. So, apparently, you do not gain any efficiency in calculating the correlation by doing that.

Suppose there is actually a trend in time in both series. This will in fact induce a correlation between the two series. You can simulate this by replacing the respective lines with:

xt <- 0 + 0.15*(1:nt) + rnorm(nt)
yt <- 0 + 0.15*(1:nt) + rnorm(nt)

Rerunning the simulation yields:

> round(apply(ri, 2, mean), 2)
[1] 0.83 0.83 0.83
> round(apply(ri, 2, sd), 2)
[1] 0.03 0.05 0.10

so all three estimators give you the same answer (on average). But pairwise deletion is actually now more efficient than using all available information from $X_t$.

Edit: You can also induce a correlation between $X_t$ and $Y_t$ without a trend, for example, by replacing the respective lines with:

xt <- 0 + 0*(1:nt) + rnorm(nt)
yt <- 0 + 0*(1:nt) + 1.0*xt + rnorm(nt)

Rerunning the simulation yields:

> round(apply(ri, 2, mean), 2)
[1] 0.7 0.7 0.7
> round(apply(ri, 2, sd), 2)
[1] 0.07 0.11 0.15

and again, pairwise deletion is more efficient.

So, I would say: No, don't do this.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I really like the idea of running a simulation. It has also been talked about in this forum about generating random numbers with a predefined correlation. It would be cool to do that, and then randomly select missing numbers and compare. (Like I think you did). (I think you just used a correlation of zero, correct?) $\endgroup$
    – Adam
    Nov 2, 2011 at 5:41
  • $\begingroup$ I simulated the data under two conditions: first when the true correlation is zero and then when the true correlation is non-zero. $\endgroup$
    – Wolfgang
    Nov 2, 2011 at 5:54
  • $\begingroup$ Sorry, you wrote " This will in fact induce a correlation between the two series. You can simulate this by replacing the respective lines with:"..I missed that somehow, thanks. $\endgroup$
    – Adam
    Nov 2, 2011 at 6:03
  • $\begingroup$ Oh, I'm bad at R - I understand it normally, but cannot write anything myself (previously I ported some code from R to Java). How can I modify it to test correlated series without trend? - Generate them as random int 1-10 (common for two series) + random value 0.0-1.0 (different for two series). (In Java this will take some minutes to write, I presume that it is easy modification to this code) $\endgroup$
    – Ivan Sopov
    Nov 2, 2011 at 9:18
  • 2
    $\begingroup$ I updated my answer to show how you can do this. $\endgroup$
    – Wolfgang
    Nov 2, 2011 at 16:46
1
$\begingroup$

I'm not sure why you want to compute the correlation of two time-series: It's generally a bad idea, especially if the two series are not stationary.

But, if they were NOT time series, but just numbers, then I would NOT use the missing data: Correlation is between pairs of values. In both versions of the formula in your link, you have sums from 1 to n for both x and y - that is not n1 and n2 or some such, just n. That implies the same thing that logic implies - don't use the missing points.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Thanks for mentioning stationarity, but I thought this to be a separate problem. $\endgroup$
    – Ivan Sopov
    Nov 1, 2011 at 12:52
  • $\begingroup$ Re stationarity; if the two series have a trend, then they will correlate because both correlate with time. For instance, manyu things about a country over time correlate with its population. Usually, though, this is not what people want: They usually want correlation "beyond" population; sometimes this can be expressed as a ratio. $\endgroup$
    – Peter Flom
    Nov 1, 2011 at 18:29
  • 1
    $\begingroup$ Actually the application in that I needed such thing have not survived the prototype phase of the life-cycle. So now I have purely theoretical interest in this question. Lets assume that two series are stationary with both mean and variance equal over time. $\endgroup$
    – Ivan Sopov
    Nov 1, 2011 at 20:42
  • $\begingroup$ @Peter I see your point that if series $X_t$ and $Y_t$ have some increasing or decreasing trend as a function of time, then this will induce a correlation between $X_t$ and $Y_t$. But I still don't see why that implies that correlating two time series is a bad idea, if what I am interested in is the correlation between the two series. Nothing about that correlation implies anything about causation. Could you elaborate why this is generally a bad idea? $\endgroup$
    – Wolfgang
    Nov 2, 2011 at 1:00
  • 1
    $\begingroup$ @Wolfgang Calculating correlations when you KNOW that there is likely a third variable that is causing both IS a bad idea. $\endgroup$
    – Peter Flom
    Nov 2, 2011 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.