my datasets have 500 variables, how to quickly verify which independent variables are significant to my dependent variable or my model? what I usually do is to import some of them, and see which one has a small p-value.

  • 2
    $\begingroup$ Try to run a lasso regression with increasing weight on the penalty terms. It performs well in variable selection task $\endgroup$ – Yang Song Mar 15 '18 at 20:59
  • 4
    $\begingroup$ What question are you trying to answer? What do you mean when you ask if they are significant? More context is needed; what makes sense in one circumstance may not in another. $\endgroup$ – Aaron - Reinstate Monica Mar 15 '18 at 21:29
  • $\begingroup$ Possible duplicate of Variable selection for predictive modeling really needed in 2016? $\endgroup$ – kjetil b halvorsen Mar 15 '18 at 21:37

It's hard to answer this without more information about your data and question, but conducting univariate tests and evaluating the p values ignores more complex intercorrelations and multivariate interactions that may be present. Using regularization during cross validation, as a commenter noted, is a more principled way to go about feature selection.


Not the answer you're looking for? Browse other questions tagged or ask your own question.