5
$\begingroup$

I'm trying to calculate time-weighted Pearson correlation as described in https://www.aaai.org/ocs/index.php/FLAIRS/FLAIRS14/paper/viewFile/7817/7840 The coefficient is given by $$ \rho_t(X,Y) = \left ( \frac{1-r}{1-r^N} \right ) \sum_{i=0}^{N} r^{i-1} \frac{(x_{-i} - \mu_X)(y_{-i} - \mu_Y)}{\sigma_X\sigma_Y}, $$ where $N+1$ is the number of observations, $t=0,1,...,N$ indicates time period when the coefficient is calculated ($t = N$ in my case, i.e. I'm calculating the correlation based on full sample), $\sigma_X$, $\mu_X$, $x_{-i}$ denote standard deviation, expectation, and the $i$th latest observation in $X$ respectively, $r$ is the decay constant, equal to a real number less than 1.

My problem is that $\rho_t(X,X) \neq 1$ when calculating it based on the formula above. Below is an example of how I calculate it in R.

set.seed(314)
N <- 1000
x <- cumsum(rnorm(N+1)) # random walk

cor(x, x) # Pearson correlation
# Result is 1

r <- 0.998
sigma <- sd(x)
mu <- mean(x)

(1 - r)/(1 - r^N) * sum( r^(0:N-1) * (x - mu)^2/sigma^2 ) # rho_t(x, x)
# Result is 1.171462

Please correct me if I have a bug in my code or suggest how I should modify the formula if it contains a mistake (my gut feeling says that the normalization might be wrong).

$\endgroup$
2
  • 1
    $\begingroup$ (+1) That paper's claims about the mathematical properties of $\rho_t$ are untrue. If you would like a formula in which $\rho_t$ is assured of lying between $-1$ and $1$, and actually equals those values when there's perfect correlation, then you also need to compute weighted versions of $\sigma_X$ and $\sigma_Y$ using the same weights. (It hardly makes sense to use unweighted estimates of standard deviations for a weighted estimate of correlation in the first place.) You might even want to use weighted estimates of the means $\mu_X$ and $\mu_Y$. $\endgroup$
    – whuber
    Apr 14, 2016 at 17:03
  • $\begingroup$ Please post your comment as an answer and I'll accept it. $\endgroup$
    – Jebus
    Apr 14, 2016 at 20:21

1 Answer 1

2
$\begingroup$

Answered in comments by whuber, copied here:

That paper's claims about the mathematical properties of $\rho_t$ are untrue. If you would like a formula in which $\rho_t$ is assured of lying between −1 and 1, and actually equals those values when there's perfect correlation, then you also need to compute weighted versions of $\sigma_X$ and $\sigma_Y$ using the same weights. (It hardly makes sense to use unweighted estimates of standard deviations for a weighted estimate of correlation in the first place.) You might even want to use weighted estimates of the means $\mu_X$ and $\mu_Y$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.