Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

I am trying to run a regression from a 11300x21500 observation matrix (where there are 11300 observations and 21500 independent variables). However, when I try to implement the usual $(X^T X)^{-1}$ formula in C++, I can't even initialize the observation matrix $X$ due to memory limitations (The application closes itself when the memory usage is around 1 GB. I have 8 GB memory but I think there is an OS command to limit memory usage of each application).

Is there any other technique which is more suitable for such a regression? Can there be any way to overcome this situation?

Thanks in advance

share|improve this question
3  
Big or small, one should (almost) never compute regression coefficients by inverting $X^T X$. Use a stable QR decomposition algorithm. (But this doesn't answer your real question.) – cardinal May 10 '12 at 12:43
4  
Also: $11300 \cdot 21500 \cdot 8 = 1\,943\,600\,000$, so even the most compact of storage is going to take up 2GB of memory just to store the matrix. And, anyway, there is no way that $X^T X$ is invertible in the first place if you have 11K observations and 21K features. – cardinal May 10 '12 at 12:50
@cardinal Before now, I was mainly working on other parts of my algorithm, and tried to avoid the invertability issue by adding small value(0.000000001) to the diagonal values of the $X^TX$ to produce a matrix that is close to the original one. But I didn't really thought on its possible objectionable consequences. I think I can find a way to work with 2GB data. But this is the first time I heard about QR decomp. Could you elaborate its usage in regressions? This isn't really my strong point. By the way, most of the columns in each row is 0 in the data.Can I take advantage of that with memory? – Osman Darin May 10 '12 at 13:24
To deal with the memory limit: open Task Manager...Processes...R-click on your stat program...Set Priority...High. – rolando2 May 10 '12 at 13:48
@rolando2: What OS is that? – cardinal May 10 '12 at 13:53
show 4 more comments

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.