# Intuition behind cross-correlation function interpretation vs. correlation of lagged time series

Can someone please explain the difference behind WHY the cross correlation function ccf() chooses to keep the same denominator for all lags and chooses to ignore the reduction in observations? Here's an example of the two methods not matching:

x = c(1,2,3,4,5,6,7,8,9,10)
y = c(3,3,3,5,5,5,5,7,7,11)
round(cor(x,y),3)
 0.896

# Think "Lag -1"
# x[-10] = 1,2,3,4,5,6,7,8,9
# y[-1] = 3,3,5,5,5,5,7,7,11
round(cor(x[-10],y[-1]),3)
 0.894

# Think "Lag -2"
# x[-10:-9] = 1,2,3,4,5,6,7,8
# y[-1:-2] = 3,5,5,5,5,7,7,11
round(cor(x[-10:-9],y[-1:-2]),3)
 0.878

print(ccf(x,y,lag.max=3))
Autocorrelations of series ‘X’, by lag

-3     -2     -1      0      1      2      3
0.197  0.466  0.699  0.896  0.436  0.221 -0.018


Notice how the Lag-0 cases matches the output of ccf(), but the negative "manual" lags do not. This is because (to my understanding) the cross correlation function will construct the "covariance" (numerator) by comparing the lagged items to the "full" 10-item mean(x) and mean(y); in addition, I believe the denominator will keep the "full" series as well.

At the end of the day, I can prove why the above Lag -1 of 0.894 does NOT match the ccf() -1 of 0.699 but I'm struggling to understand WHY the ccf() functions chooses to do what it does?

I'm guessing it has something to do with adjusting for some sort of bias...?

The problem is not the normalisation constant, since in correlation formula it simply cancels out. The difference arises because means and variances of the series are held fixed when calculating the cross-correlations. This means that variance and means are calculated for the whole series, and they are used in calculating correlation when the length of series decreases due to lags. This is a perfectly valid operation if the series are considered stationary, i.e. with constant mean and variance.

Here is the detailed example which recreates the behaviour of ccf:

x = c(1,2,3,4,5,6,7,8,9,10)
y = c(3,3,3,5,5,5,5,7,7,11)

mx <- mean(x)
my <- mean(y)
dx <- mean((x-mx)^2)
dy <- mean((y-my)^2)
nx <- length(x)

round(cor(x,y),3)
 0.896

cr<-function(x,y,mux=mean(x),muy=mean(y),dx=var(x),dy=var(y),n=length(x)) {
cxy<-sum((x-mux)*(y-muy))/n
cxy/sqrt(dx*dy)
}
round(cr(x,y,mx,my,dx,dy,nx),3)
 0.896

# Think "Lag -1"
# x[-10] = 1,2,3,4,5,6,7,8,9
# y[-1] = 3,3,5,5,5,5,7,7,11
round(cor(x[-10],y[-1]),3)
 0.894
round(cr(x[-10],y[-1],mx,my,dx,dy,nx),3)
 0.699
# Think "Lag -2"
# x[-10:-9] = 1,2,3,4,5,6,7,8
# y[-1:-2] = 3,5,5,5,5,7,7,11
round(cor(x[-10:-9],y[-1:-2]),3)
 0.878
round(cr(x[-10:-9],y[-1:-2],mx,my,dx,dy,nx),3)
 0.466

print(ccf(x,y,lag.max=3,plot=FALSE))

Autocorrelations of series ‘X’, by lag

-3     -2     -1      0      1      2      3
0.197  0.466  0.699  0.896  0.436  0.221 -0.018


Note that the norming constant in the function cr is needed only because it must be the same norming constant used in the variance calculations.

• This is helpful and does an excellent job illustrating the differences between the methods; however, can you elaborate on your comment? "...perfectly valid oepration if the series are considered stationary" Specifically, when would it be advantageous to NOT holding the means and variances fixed? Similarly, is it always best practice to use CCF() when the series are in fact stationary or is there a case to be made for NOT FIXING the means and variances in certain situations? – JasonAizkalns Oct 3 '13 at 12:56
• If the series are non-stationary then sample autocorrelations are meaningless, i.e. they are spurious (it is probably possible to construct non-stationary processes for which they are not spurious). So there is no advantage in not holding means and variances fixed. – mpiktas Oct 3 '13 at 13:06
• I have study where I correlate many pairs of time series. it happens often that the r at lag 0 taken by ccf() and the r from the simple cor() are different. any ideas why? the time series has NAs and ccf() is used with na.action=na.pass and the cor() use "pairwise.complete.obs". could this be the cue? – Bakaburg Apr 9 '15 at 16:27
• At lag zero correlation should be 1. – mpiktas Apr 10 '15 at 5:58