Calculating Regression Coefficients for Very Large Observation Matrices

I am trying to run a regression from a 11300x21500 observation matrix (where there are 11300 observations and 21500 independent variables). However, when I try to implement the usual $(X^T X)^{-1}$ formula in C++, I can't even initialize the observation matrix $X$ due to memory limitations (The application closes itself when the memory usage is around 1 GB. I have 8 GB memory but I think there is an OS command to limit memory usage of each application).

Is there any other technique which is more suitable for such a regression? Can there be any way to overcome this situation?

Big or small, one should (almost) never compute regression coefficients by inverting $X^T X$. Use a stable QR decomposition algorithm. (But this doesn't answer your real question.) –  cardinal May 10 '12 at 12:43
Also: $11300 \cdot 21500 \cdot 8 = 1\,943\,600\,000$, so even the most compact of storage is going to take up 2GB of memory just to store the matrix. And, anyway, there is no way that $X^T X$ is invertible in the first place if you have 11K observations and 21K features. –  cardinal May 10 '12 at 12:50
@cardinal Before now, I was mainly working on other parts of my algorithm, and tried to avoid the invertability issue by adding small value(0.000000001) to the diagonal values of the $X^TX$ to produce a matrix that is close to the original one. But I didn't really thought on its possible objectionable consequences. I think I can find a way to work with 2GB data. But this is the first time I heard about QR decomp. Could you elaborate its usage in regressions? This isn't really my strong point. By the way, most of the columns in each row is 0 in the data.Can I take advantage of that with memory? –  Osman Darin May 10 '12 at 13:24