Is there a way in which MARS/EARTH can work with ff objects or big data?

I need an additive model for a sample of size 20000 and around 100 variables. I can not work with that dimension with a RAM so I need to find another option to work with big data.


Earth will easily handle your 20 thousand by 100 matrix. That model requires less than a quarter of a gig of memory.

The inst/slowtests/test.big.R file in the earth test suite has a model with 6 million by 100 matrix (tested on a machine with 32 GB of memory).

Earth doesn't have facilities for ff (and similar) objects, because the limit in practice is model building time rather than memory.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.