1
$\begingroup$

I am working on this problem where I have 20 odd features (input variables) and two dependent variables. The objective is to find the variance structure of one of the dependent variables. More importantly, I need to find which of the features explain the most amount of variance in the output/dependent variable. I had the following 3 approaches in mind:

  1. Run a multiple regression on the standardised input variables and see which one has the highest coefficient. Would this tell me that the feature with the highest regression coefficient explains the most amount of variance in the output variable?
  2. I read some research papers on the decomposition of R-squared, where the statistical software would tell us how the R-squared of the regression is decomposed into various features. I don't know how to implement this on Python or any of its libraries. Could someone help?
  3. I know that PCA is used for feature selection but I am not completely sure how to get the original features that explain the most amount of variance in the dependent variable using PCA. Does PCA only tell me dimensions along which the data has the highest amount of variance or does it specifically tell me dimensions, and by extension, original features, that explain the variance in the dependent variable? Could anyone guide me on this?

What approach, if any other, should I be following with regards the above objective?

$\endgroup$
0
1
$\begingroup$

Unless you have a very high signal:noise ratio or an enormous sample size, the data are incapable of reliably informing you of which variables are "doing the job". The bootstrap will document the difficulty of the task. A complete example is found in my RMS course notes Section 5.4 where the bootstrap obtains confidence limits for the ranks of variable importance measures. You can also use the bootstrap to get confidence intervals on the importance measures themselves, just as easily. Still, we often summarize our best guess at variable importance using dot charts to show the partial $\chi^2$ statistics, for which you'll see many examples in the course notes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.