Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite

We are trying to experiment using the gbm package on a quite large dataset (~140 million rows) and we have ran into a problem with the memory requirements of R.

We have tried combining the packages 'gbm' and 'bigmemory' with no success and our next thought was to modify the C++ source code to draw data from a local database where we have stored our dataset.

So, we were wondering if there is a more appropriate or well-known practice in order to change the allocation inside the C++ code of gbm. Has anyone tried something similar?

In http://stackoverflow.com they have recommended to use sampling, but in this initial step we would like to pass all these data if it is possible.

share|improve this question
While you are contemplating a purely software solution, this question is-on topic at SO but off-topic here. If you change your mind and consider a sampling-based alternative, then the specific question of how to sample and how to interpret the results would be welcome here. – whuber♦ Jul 27 '12 at 15:22

closed as off topic by whuber♦ Jul 27 '12 at 15:22

Questions on Cross Validated are expected to relate to statistics within the scope defined in the FAQ. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about closed questions here.

Browse other questions tagged or ask your own question.