3
$\begingroup$

Say I have 1000 potential predictors in a logistic regression. I don't have time to check each predictor manually for linearity. I could wait till after variable selection, but in that case I wonder if some predictors that have strongly non-linear relationship with the mean may not be selected because their non-linear relationship confounds their significance. suppose logit(P(Y=1))=B_0+B_1X_1+B_2g(X_2) for some unknown g, but I regress logit(P(Y=1))=b_0+b_1X_1+b_2X_2, could it be the case that B_2<>0 but b_2 is not significant? Prhaps because SE(b_2) is too large due to the poor assumption that g(x)= x? Or what if there is collinearity between X_1 and X_2 enough that b_2 appears not significant but there is less collinearity between X_1 and g(X_2)? If this happens I would not know that some of the rejected predictors are actually useful, and then I would be back to needing to find some way to check for appropriate transformations automatically. Is there a good way to automatically check for tranformations when there are hundreds of predictors?

$\endgroup$
  • $\begingroup$ "$P(Y=1)=B_0+B_1X_1+B_2X_2$" is not logistic. $\endgroup$ – Glen_b Oct 2 '14 at 20:11
  • $\begingroup$ Of course P(Y=1) in the above should be replaced by logit(P(Y=1)) $\endgroup$ – takintayo akinbiyi Oct 3 '14 at 16:10
  • $\begingroup$ Please do so, via edits to your equations. $\endgroup$ – Glen_b Oct 3 '14 at 16:12
2
$\begingroup$

This would be a good candidate for either a lasso or elastic net regression methodology, which perform variable selection as well as coefficient shrinkage for collinear variables. Include the main effect terms, plus quadratic (or higher power) terms to account for possible nonlinear relationships.

$\endgroup$
  • $\begingroup$ In R, you could use glmnet package written by the inventors of lasso to cut down on the number of variables. You will still have to do all your own transformations unfortunately. Or you can use svm on the factors left after glmnet lasso (alpha = 1) does it doing to transform them into infinite space. $\endgroup$ – ignorant Oct 3 '14 at 18:45
0
$\begingroup$

You can check out the maximal information coefficient statistic. It will compute the dependencies (both linear and nonlinear) pairwise among features and it can be very convenient for large numbers of features.

Here is a link to the website. http://www.exploredata.net/

I believe recently an R package called minerva was also released which computes this statistic for you.

$\endgroup$
  • $\begingroup$ I would be interested to read an expanded version of this answer. $\endgroup$ – rolando2 Feb 26 '15 at 16:17
0
$\begingroup$

Compute the eigenvalues and eigenvectors of X'X. Eigenvalues that are close to 0 will correspond to eigenvectors that represent a set of multicollinear columns.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.