I have a very large scale tabular dataset that have more than 200 features and more than 2 million records. I would like to run some machine learning models to predict some target $y$. However, it is difficult to put even 70% of the data (I was expecting 95% or 98%) into the memory no matter how I optimize my code (I am using Julia).

Except going on optimizing my code and trying to find some good libraries, I am now thinking maybe it is a good idea to look at the data itself.

I believe there are some redundancies within the data, there are two thoughts

  • From the perspective of mapping $f:\mathbf{x}\mapsto y$. Not all features are predictive of $y$. Therefore, these features could be removed. A thread mentions this and points to a library to do this.
  • From the perspective of feature $\mathbf{x}$. Some of the features could be predicted from others and these features are proxies. If we could somehow effectively remove proxies. Then hopefully we could largely reduce the number of features. I haven't seen people do this.

My question is: does the second thought look promising? If it does, how could I implement it?

  • $\begingroup$ Your second point requires making 200 models. For anything more complex than a linear regression, this may be prohibitive, in which case just remove highly correlated features. $\endgroup$ – Demetri Pananos Jan 16 at 1:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.