Cross-correlation of massive arrays I have 16 1D arrays with approximately 10-11 million double-precision elements each. I need to execute a cross-correlation across them, i.e., 1 with 2, 1 with 3, ..., 1 with 16, 2 with 3, 2 with 4, ..., 2 with 16, and so on. This cannot be done efficiently on my MacBook Intel Core 2 duo 2.4 GHz, with 4GB of RAM. My question is, what is the typical approach, if not brute force (faster processor, more RAM) that people use to overcome this problem, or problems like it?
I asked this question on Stack Overflow a few minutes ago, but I realized that there might be more people here who have dealt with this particular problem.
 A: I'm surprised this isn't possible, actually. I'm running a similar setup to you (8GB instead of 4, but I doubt that makes a difference here) and I see (using R here):
> x=runif(1e7)
> y=runif(1e7)
> system.time(cor(x,y))
   user  system elapsed 
  0.129   0.001   0.128 

So for (16 choose 2) = 120 of these computations, that'd be around 15 seconds. Are you perhaps computing the correlations via an inefficient loop in whatever language you're using, instead of using the built-in function? Or is the computation you're doing more complicated for some reason?
EDIT: storing the values as columns of a matrix, the computation is even simpler and faster:
> X = matrix(runif(16*1e7), ncol=16)
> system.time(cor(X))
   user  system elapsed 
  3.684   0.003   3.687 

A: Consider using multiple cores: the snow package allows you to build a cluster on the same machine or on multiple computers in your network. Then you can use parLapply() to distribute the cor() function. It works exactly like lapply() except it will send each cor() call to a different process in the cluster.
