my datasets have 500 variables, how to quickly verify which independent variables are significant to my dependent variable or my model? what I usually do is to import some of them, and see which one has a small p-value.


closed as unclear what you're asking by Stephan Kolassa, kjetil b halvorsen, Michael Chernick, mdewey, Peter Flom Mar 16 '18 at 14:38

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    $\begingroup$ Try to run a lasso regression with increasing weight on the penalty terms. It performs well in variable selection task $\endgroup$ – Yang Song Mar 15 '18 at 20:59
  • 4
    $\begingroup$ What question are you trying to answer? What do you mean when you ask if they are significant? More context is needed; what makes sense in one circumstance may not in another. $\endgroup$ – Aaron Mar 15 '18 at 21:29
  • $\begingroup$ Possible duplicate of Variable selection for predictive modeling really needed in 2016? $\endgroup$ – kjetil b halvorsen Mar 15 '18 at 21:37

It's hard to answer this without more information about your data and question, but conducting univariate tests and evaluating the p values ignores more complex intercorrelations and multivariate interactions that may be present. Using regularization during cross validation, as a commenter noted, is a more principled way to go about feature selection.


Not the answer you're looking for? Browse other questions tagged or ask your own question.