I'm analyzing a large problem with a large $N \times M$ data matrix $A$, where $N$ is the number of observations, $M$ is the number of explanatory variables, and $N \gg M$. I'd like to perform single-pass linear regression on this data set against a scalar response variable. But the challenge is that I'm only able to load $N \times 1$ columns of $A$ at a time (and loading row-by-row is not possible for this problem).
So the question is, is there some way to compute $A^T A$ for linear regression using less than $O(NM)$ memory by loading columns of $A$ at a time?
Edit: Thanks for your interest. I'm looking for an efficient algorithm that makes a single pass through the data. I'd like to compute $A^T A$ exactly if possible.
Each column contains 200 million rows and is stored in separate compressed files that disallow partial read. Production system needs the data in this format. There are 60 thousand columns. The data is on a storage server, and it takes 10 seconds to load one column on a compute server in the same data center and a week to make a single pass through the data. Each compute server has 1 TB of memory. I'm currently working in parallel batches of 2 M rows. Finally, it is not possible to duplicate the data for analytics.