Technique to find spurious correlations among huge number of time series datasets?

Question

I came across this tongue-in-cheek website that lists lots of spurious correlations. It's not lost on me that the author's main point is to discourage the brute-force-search of correlations, but his website has encouraged me to do the same (but with plausible examples) at my employer. My job is to discover new insights (but not cause-effect relationships per se) in our hundreds of thousands of financial time series data sets, and I'm hoping this approach will make me a more efficient employee.

Is there a name associated with this brute force approach? I would like to research it further. Aside from the obvious requirement of thorough follow-up / vetting any of potential discoveries, what other practical or implementation considerations should I be aware of? Are there any open source (e.g., Python or R) libraries that make this brute force search easier?

Answer 1

For contemporaneous correlation I'd go with PCA. You look at the eigen vectors which have some components dominating the others in absolute values. However, this will not be able to catch the phase shifted correlations well. For instance, if you have a sine and cosine waves of the same frequency then correlation is going to be near zero, despite cosine being a sine with a phase shift. Still, if the phase shift is not too close to $\pi/2$, then this should catch the correlations to some degree.

To handle the phase shifted (lagged) correlations the first thing that comes to my mind is coherence measure (related to cross-correlation and cross-spectra density). So, for the above example of sine and cosine waves, coherence will show you perfect 1, it'll see that it's the same wave. You can additionally get the phase shift.

This is not the most efficient way because it requires pair wise analysis, i.e. the number of calculations grows as $n^2$ with $n$ number of series. It's certainly a brute force type of approach.

Another way is by running FFT on series, then comparing the spectral densities. This is a bit tricky because you need to come up with a measure of similarity of densities. You could go for the biggest peaks and see whether they're close to each other to some degree. Suppose, you have two series with both having the biggest peaks in terms amplitudes in spectra at 10 days cycle, then they're correlated with some lag, that can also be obtained by looking at the phase component of FFT