I was wondering what are some techniques for variable selection when there are a large number of variables lets say 1000, and the entire dataset is too large to fit into memory. How would one go about doing variable selection without loading the entire dataset?

  • $\begingroup$ This would depend on the package you are using... $\endgroup$ – Marquis de Carabas Aug 12 '15 at 22:43
  • $\begingroup$ I was just wondering in a general sense. $\endgroup$ – stochasticcrap Aug 12 '15 at 22:45
  • $\begingroup$ I always encourage people to start with variables of theoretical importance and go from there. The literature should tell you the variables you might want to keep in your dataset. As far as the actual codes/commands to keep certain variables--well that depends on the statistical package. In Stata, you can specify variables you want to pull into memory with the following syntax: use [varlist] using "blablabla.dta". Not an R user, so I don't know how to do it in R. I haven't had to do something like that in SAS... $\endgroup$ – Marquis de Carabas Aug 12 '15 at 22:48
  • $\begingroup$ It depends on nature of variables and source. In chemical process industry we usually use averages or moving averages. By this way the data size is reduced and it helps us to use. $\endgroup$ – iskbaloch Mar 22 '19 at 8:35
  • $\begingroup$ Welcome to CV. The answer is very incomplete. What averages are you proposing? How do you deciede what averages to take? What is the theoretical justification for the approach? When is it applicable, when is it not? $\endgroup$ – ReneBt Mar 22 '19 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.