4
$\begingroup$

In addition to proc varclus, randomForest, and assessing multicollinearity among potential predictor variables, I am seeking other methods of variable selection in lieu of using stepwise methods for building more parsimonious binary logistic regression models from a wide array of potential predictor variables. I have done some research into other methods such as Mutual Information (MI), and I have two questions in regards to its use:

1) Has anyone used MI for binary logistic regression variable selection? If so, what are your thoughts on its application?

2) Does anyone know how to calculate MI using either Base SAS or R for potential predictor variables in reference to the outcome of interest? Any help or references in this area would be greatly appreciated!

Thanks!

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ are you looking for continuous or categorical predictor variables? see stats.stackexchange.com/questions/68035/… - G-test en.wikipedia.org/wiki/G-test $\endgroup$
    – seanv507
    Commented Aug 23, 2013 at 7:11
  • $\begingroup$ @seanv507, both, but good question, I am aware that MI is used for two categorical variables, but I was wondering how useful it is and how to apply it within SAS and/or R. Thanks! $\endgroup$
    – Matt Reichenbach
    Commented Aug 23, 2013 at 11:15

1 Answer 1

Reset to default
1
$\begingroup$

As I expected, no one probably uses MI to perform variable selection, and thus I will stick with other methods such as variable clustering, random forest, PCA, correlation assessment among potential predictor variables, and other methods in lieu of using MI to deal with high dimensionality.

Additionally, MI is likely not that useful for continuous variables, which encompass a large percentage of the candidate predictor variables in a typical dataset that I work with, so I am likely much better off without making using of MI as another variable selection technique even if it is a worthwhile approach to do so with continuous potential predictor variables.

PLEASE correct me if I am wrong, as I would enjoy exploring a new technique (at least to me) of avoiding the use of stepwise techniques to reduce the number of candidate variables in my logistic regression models!

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ MI is often used in feature selection. Try to have a look at the minimum redundancy maximum relevancy method (mRMR). In case of continuous variables they go for discretization. You could also use an estimator of MI for continuous feature but in mRMR paper they say that dicretization led to better results. Another interesting method for feature selection is the one in Boruta package. As output you will get a set of potential predictor variables useful to predict the class. However, they might be correlated to each others. $\endgroup$
    – Simone
    Commented Sep 5, 2013 at 4:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.