# Importance of variables in logistic regression

I am probably dealing with a problem that has probably been solved a hundred times before, but I'm not sure where to find the answer.

When using logistic regression, given many features $x_1,...,x_n$ and trying to predict a binary categorical value $y$, I am interested in selecting a subset of the features which predicts $y$ well.

Is there a procedure similar to the lasso that can be used? (I have only seen the lasso used for linear regression.)

Is looking at the coefficients of the fitted model indicative of the importance of the different features?

# Edit - Clarifications After Seeing Some of the Answers:

1. When I refer to the magnitude of the fitted coefficients, I mean those which are fitted to normalized (mean 0 and variance 1) features. Otherwise, as @probabilityislogic pointed out, 1000x would appear less important than x.

2. I am not interested in simply finding the best k-subset (as @Davide was offering), but rather weigh the importance of different features relative to each other. For example, one feature might be "age", and the other feature "age>30". Their incremental importance might be little, but both may be important.

Thank you.

-

The answer to your last question is a flat NO. The magnitude of coefficients are in no way a measure of importance. The lasso can be used for logistic regression. You need to study the area more assiduously. The methods you need to study are those that involve "penalized" methods. If you are looking for detection methods that uncover "shadowed" predictors, a term that may be defined somewhere but is not in general use, then you need to be looking for methods that inspect interactions and non-linear structure within the predictor space and the outcome linkage to that space. There is quite a bit of discussion of these issues and methods in Frank Harrell's text "Regression Modeling Strategies".

The backward selection strategy will fail to deliver valid results (although it does deliver results). If you looked at a case of 20 random predictors for 100 events you will probably find 2 or 3 that will be selected with a backward selection process. The prevalence of backward selection in the real world reflects not careful statistical thought but rather its easy availability in SAS and SPSS and lack of sophistication of those products' user base. The R user base has a harder time accessing such methods and users that post requests on the mailing lists and SO they generally get advised of the problems involved with backward (or forward) selection methods.

-
I know that I should - I would greatly appreciate some pointers as to where to start. –  Guy Adini Jan 6 '13 at 5:29
I'll add an example to back up this one. Suppose we set $x_{n+1}=1000x_{1}$. Then the (unpenalised) estimated coefficient for $x_{n+1}$ will be $1000$ times smaller than the (unpenalised) estimated coefficient for $x_{1}$. But notice that the strength of the two predictors will be exactly the same. –  probabilityislogic Jan 6 '13 at 14:09
Please see my comments above (using normalized features). Thanks. –  Guy Adini Jan 6 '13 at 16:59
Thank you. I will look into that. Can you name a few common algorithms that are used in this " inspection of interactions and non-linear structure within the predictor space", or is it a very case-by-case situation? –  Guy Adini Jan 6 '13 at 20:39
You can use regression splines to search for non-linearity and spline terms can be "crossed", which allows identification of effects that are restricted to one region of a 2D prediction space. You can also use local regression methods. In R the most used local regression method is probably the 'mgcv' package, but the older 'locfit' package is still available. –  DWin Jan 6 '13 at 20:52