In the following blog post: here, the author goes and calculates the alternative correlation measure for two time series. Essentially, calculates the correlation between two series by giving the mean of each series 0. I tried reproducing it, but am not getting close results.

Here is the code:

con = gzcon(url('http://www.systematicportfolio.com/sit.gz', 'rb'))

tickers = spl('SPY,GLD')

data <- new.env()
getSymbols(tickers, src = 'yahoo', from = '1970-01-01', env = data, auto.assign = T)
for(i in ls(data)) data[[i]] = adjustOHLC(data[[i]], use.Adjusted=T)        
bt.prep(data, align='remove.na', dates='1994::')

prices = data$prices["2012:10::",] #use dates in line with post
sma<-bt.apply.matrix(prices,SMA,40) #moving Average

osc<-na.omit((prices - sma) / sma) #percent deviation from MA


mean.x<-0    #0 #mean(x)
mean.y<-0    #0 #mean(y)


sum(((x-mean.x)/(sd.x))*((y-mean.y)/(sd.y))) / (nrow(x)-1) #equation for correlation

In the blog post, the author showed 2 example results. One used a MA length of 20 days and another used an MA length of 40 days. I am using the following correlation formula:

enter image description here

I get values of -17 and -1.8 for 20 day MA and 40 day MA respectively. This differs from-9 and -54 from what the author used. Any ideas as to what I am doing incorrectly?

  • $\begingroup$ Just asking, but why are you sure that the blogger got the right answer? $\endgroup$ – Placidia May 23 '13 at 1:12
  • $\begingroup$ I don't actually. I just thought I might get different interpretation of measuring correlation. I'd love to hear feedbacks if possible about alternative correlation measures if possible $\endgroup$ – user1234440 May 23 '13 at 13:41

OK. Well, if you are looking for alternative correlation measures, maybe we should start with some basic time series stuff - which the author of this blog seems unaware of.

Of course, getting a quantitative measure of “highly correlated” to match with my intuitive understanding has never worked out so well. Correlated, for me, tends to mean: “oscillates with the same periodicity in the same overall trend.” A daily correlation measure, as defined mathematically, is actually a measure of how divergent two time-series’s noise is (where noise is the divergence of a time-series from its average trend). These definitions certainly don’t line up.

Hoffstein's intuition of correlation corresponds fairly closely to my intuition of what a spectral decomposition is. Spectral decomposition ... and I'm glad you asked ... represents how closely a series resembles a sine/cosine function of given periodicity. It's derived from the Fourier transform of the series. This is called a frequency domain analysis.

Correlation is related to the extent to which two entities are linearly related. Correlations belong to the so-called time domain analysis.

When you look at an individual time series, you look at its correlation with itself at previous time points - that's the auto-correlation function.

The cross-correlation function looks at how $X(t)$ relates to $Y(t-L)$ for lags $L$. Hoffstein seems to be shooting for the cross-correlation of lag 0, but he misses. All of these series that look at correlations (or spectra) assume that the time series have mean 0. That means that if they don't have mean 0, you subtract the mean. And furthermore, if there is a fixed linear trend, you estimate it and remove it. Otherwise, as he points out, you are confounding the trend effect and the stochastic effect.

There's more. The sample auto-correlations only converge if the relationship between subsequent values is sufficiently small - and the correlations have to dampen out with time. The estimates don't converge with something like a random walk $X_t=X_{t-1}+\epsilon_t$, so the first thing you do (if you suspect random walk type models) is take the first differences of the series and analyse those. If Hoffstein took the first differences of the two plots he has, the remainders would probably be ARMA (auto-regressive moving average processes), and he could sensibly look at correlations and cross-correlations.

I am trying to cram a whole course in time series into one posting. The point is that series like Hoffstein's can arise from two (at least) models. You can assume a linear function of time with random (correlated) error - or you can assume a fully stochastic model (like random walk). There are fully stochastic models whose trajectories mimic deterministic models fairly well. Either way, a straight-up autocorrelation or cross-correlation will not work.

You can see from your own workings, and Hoffstein's, that your correlations are seriously not converging. I don't know from your code why you got a different answer from H. - I am not familiar with the functions in quantmod --- but you can see that whatever it is, it's a bad idea. I think the problem is that $\bar{x}$ and $\bar{y}$ are not converging, since they are being taken over trending time frames.

Hoffstein says

We’ve blogged about how it can be a deceiving metric before, but never offered up more of a solution than simply, “make sure you assume a zero-mean.”

Well, no. It's not important to assume a zero mean. It's important to have a zero mean - and if you need to difference your series to get it, then that's what you do.

There is more wrong with Hoffstein. Why does he take moving averages of his series before looking at correlations? The moving average induces autocorrelations into the series. If the series had been white noise prior to his manipulations, they would end up with aucorrelations after the fact. This is called the Slutsky-Yule effect. I don't understand what he is doing with his "% difference from price - whether this is a % difference within the same series (Hello, Messers. Slutsky and Yule), or if he is taking the % of one series again the other (which would induce cross-correlations) --- but it's messy and horrible and I don't like it.

The starting point has to be the model - but I am old-fashioned. Ask yourself what you know about the data; ask yourself what you would like to know; then build a model and estimate the parameters. Otherwise, it looks a lot like puddling, although one might call it exploratory data analysis if one wanted to put a good face on it.

If Hoffstein wants to relate SPY and GLD, he could possibly just regress one on the other - perhaps with some iterative process for the errors (which would be correlated). You would need a weighted least squares to account for the error structure - but there are ways of doing that.


It seems an off-the-cuff blog post has run wild here, so please afford me the opportunity to respond.

[Edited shortly after original post] In recreating my results, it looks like there are potentially two pieces missing. Firstly, the standard deviation measure must also assume a zero mean for the distribution (so the square-root of the mean of squared daily distance measures). Secondly, the sensitivity of the similarity measure is going to be based on the moving averages used and the length of the measurement period. Extending the data from 5/15/2013 to 6/18/2013 changed the measure from -9% to -17% with the 20 DMA model.

I don't quite know why Placidia seemed to so aggressively attack my understanding of time-series; nowhere in my post do I present this alternative measure as a legitimate, mathematically sound model. Perhaps I should have more concretely not stated it as such.

She is absolutely correct in her definitions of spectral decomposition, linear correlation, and autocorrelation. I was not shooting for a lagged cross-correlation function (if I was, I would have used it). I do not contest that the model I presented is mathematically unsound.

But there is a big difference between mathematically accurate and pragmatically useful. I could have differenced the two models and tried to fit an ARMA(p,q) model. But what are p and q? Then we have to start talking AIC and BIC to find the most parsimonious, but statistically significant model. Then I have to worry whether I can assume the relationship will remain stationary long enough that I can continue to use this model going forward.

In this particular case, that solution was overkill. I am a big believer that simple and intuitive is more robust than complicated -- even if complicated is more purely mathematically accurate. My intention was not to say that the measure I was playing with was better, but simply, in my case, potentially more useful for what I was trying to capture.

She says that having a zero mean is important and you should difference to get there. This is absolutely, mathematically true. Looking at financial data, however, we see that long-term, daily return distributions tend to have a very near-zero mean return -- so the point is moot. In the short-term, however, securities can exhibit statistically significant trends. By assuming a zero mean, rather than differencing out to a zero mean, we incorporate this trend within our data and therefore our correlation measure. A quick and dirty trick, but it works for us because we know we're doing it.

I can appreciate she doesn't like it. It's non-pristine mathematics. It's messy. It has warts and bumps and doesn't fit in the right, rigid boxes. But it is pragmatic for me as a quick measure that helps incorporate the short-term, statistically-significant trends that can occur in financial returns, into a simple, quick, off-the-cuff similarity measure.

  • $\begingroup$ ...it is entirely possible that the results are wrong. Can't you check? Can you show us code? $\endgroup$ – Nick Cox Jun 19 '13 at 16:14
  • $\begingroup$ I had original done the work in a quick excel file before translating to Python. I recreated in Excel and noticed a couple points which I edited into my response above. $\endgroup$ – Corey Hoffstein Jun 19 '13 at 17:35
  • 1
    $\begingroup$ IMA: Your response seems unnecessarily aggressive and accusatory for, again, what was just intended as an off-the-cuff toy model that I was opening to my readership for discussion as a new, pragmatic way to help capture the trend component into a similarity metric without getting into more complex time-series methodologies. I never tried to objectively define "better" or "faster" because the model was merely a sketch and not meant as a comparison to anything. The goal was simply to create a similarity metric that captured trend, which was how I was measuring "sufficient". $\endgroup$ – Corey Hoffstein Jun 19 '13 at 18:17
  • 1
    $\begingroup$ Nick: Reproducible code has been put on the original blog post with an edit at the top linking to this discussion (for transparency) but also to clarify that it is merely an off-the-cuff toy model. $\endgroup$ – Corey Hoffstein Jun 19 '13 at 18:33
  • 1
    $\begingroup$ I posted this question as a means to get some help from the general public. I did not intend this to be a criticism on anyones work. I work in the same industry and sometimes the profitable and meaningful ideas come simply from things that don't make sense in academia. Corey has to all intent and purposes offered full transparency and I really appreciate that. $\endgroup$ – user1234440 Jun 19 '13 at 19:45

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.