I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y?

How does one decide where to start from? Reduce a large number of columns without the benefit of domain knowledge? Basic stats like removing - large number of null columns , single values aside what other steps do data scientists usually use ?

  • $\begingroup$ 200 variables given your sample size is not that many. What you want: explanatory or predictive model? You can simply use regularized regression. $\endgroup$ – Tim Jan 29 at 5:57
  • $\begingroup$ How does regularized expression help in reducing the manual work of going through 200 columns ? $\endgroup$ – Adurthi Ashwin Swarup Jan 29 at 7:27
  • $\begingroup$ Regularization does automatic feature selection stats.stackexchange.com/q/4272/35989 $\endgroup$ – Tim Jan 29 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.