I am probably dealing with a problem that has probably been solved a hundred times before, but I'm not sure where to find the answer.

When using logistic regression, given many features $x_1,...,x_n$ and trying to predict a binary categorical value $y$, I am interested in selecting a subset of the features which predicts $y$ well.

Is there a procedure similar to the lasso that can be used? (I have only seen the lasso used for linear regression.)

Is looking at the coefficients of the fitted model indicative of the importance of the different features?

Edit - Clarifications After Seeing Some of the Answers:

  1. When I refer to the magnitude of the fitted coefficients, I mean those which are fitted to normalized (mean 0 and variance 1) features. Otherwise, as @probabilityislogic pointed out, 1000x would appear less important than x.

  2. I am not interested in simply finding the best k-subset (as @Davide was offering), but rather weigh the importance of different features relative to each other. For example, one feature might be "age", and the other feature "age>30". Their incremental importance might be little, but both may be important.


4 Answers 4


DWin's response offers the answer but little insight, so I thought it might be useful to provide some explanation.

If you have two classes you are basically trying to estimate $p=P(y_i=1|X=x_i)$. This is all you need and logistic regression model assumes that:

$log \frac{p}{1-p} = log \frac{P(y_i=1|X=x_i)}{P(y_i=0|X=x_i)}=\beta _0 + \beta _1 ^T x_i$

What I think you mean by the importance of the feature $j$ is how it affects $p$ or in other words what is $\frac{\partial p}{\partial x_{ij}}$.

After a small transformation you can see that

$p=\frac{e^{\beta _0 + \beta _1 ^T x_i}}{1+e^{\beta _0 + \beta _1 ^T x_i}}$.

Once you calculate your derivative you'll see that

$\frac{\partial p}{\partial x_{ij}} = \beta_j e^{\beta_0 + \beta _1 ^T x_i}$

This clearly depend on the value of all other variables. However you can observe that the SIGN of the coefficient can be interpreted the way you want: if it is negative then this feature decreases the probability p.

Now in your estimation procedure you are trying to estimate $\beta$s assuming your model is correct. With regularization you introduce some bias into these estimates. For a ridge regression and independent variables you can get an closed form solution:

$\hat{\beta^r} = \frac{\hat{\beta}}{\hat{\beta} + \lambda}$.

As you can see this can change the sign of your coefficient so even that interpretation break apart.

  • 1
    $\begingroup$ typo in the denominator of eq1? $\endgroup$
    – Fernando
    Commented Dec 13, 2016 at 14:56

The answer to your last question is a flat NO. The magnitude of coefficients are in no way a measure of importance. The lasso can be used for logistic regression. You need to study the area more assiduously. The methods you need to study are those that involve "penalized" methods. If you are looking for detection methods that uncover "shadowed" predictors, a term that may be defined somewhere but is not in general use, then you need to be looking for methods that inspect interactions and non-linear structure within the predictor space and the outcome linkage to that space. There is quite a bit of discussion of these issues and methods in Frank Harrell's text "Regression Modeling Strategies".

The backward selection strategy will fail to deliver valid results (although it does deliver results). If you looked at a case of 20 random predictors for 100 events you will probably find 2 or 3 that will be selected with a backward selection process. The prevalence of backward selection in the real world reflects not careful statistical thought but rather its easy availability in SAS and SPSS and lack of sophistication of those products' user base. The R user base has a harder time accessing such methods and users that post requests on the mailing lists and SO they generally get advised of the problems involved with backward (or forward) selection methods.

  • 2
    $\begingroup$ I know that I should - I would greatly appreciate some pointers as to where to start. $\endgroup$
    – Guy Adini
    Commented Jan 6, 2013 at 5:29
  • $\begingroup$ I'll add an example to back up this one. Suppose we set $x_{n+1}=1000x_{1}$. Then the (unpenalised) estimated coefficient for $x_{n+1}$ will be $1000$ times smaller than the (unpenalised) estimated coefficient for $x_{1}$. But notice that the strength of the two predictors will be exactly the same. $\endgroup$ Commented Jan 6, 2013 at 14:09
  • $\begingroup$ Please see my comments above (using normalized features). Thanks. $\endgroup$
    – Guy Adini
    Commented Jan 6, 2013 at 16:59
  • $\begingroup$ Thank you. I will look into that. Can you name a few common algorithms that are used in this " inspection of interactions and non-linear structure within the predictor space", or is it a very case-by-case situation? $\endgroup$
    – Guy Adini
    Commented Jan 6, 2013 at 20:39
  • $\begingroup$ You can use regression splines to search for non-linearity and spline terms can be "crossed", which allows identification of effects that are restricted to one region of a 2D prediction space. You can also use local regression methods. In R the most used local regression method is probably the 'mgcv' package, but the older 'locfit' package is still available. $\endgroup$
    – DWin
    Commented Jan 6, 2013 at 20:52

I am interested in selecting a subset of the features which predicts y well.

this means: you need existance of Dependance between X & Y. And this dependance can easily be revealed with scatter-plot if dependancy exists (even with OLS-regression line at each subchart). Be careful: sometimes some feature engineering transformations could be needed to reveal this dependance -- thus way the whole process of Feature Selecton usually starts from Explorative Data Analysis (you can even search for EDA in one line libraries if they suits your data analysis).

this also means statistically: that you need to find features (x) that explain variance of your target_data (as is y) -- if you need this - you can use Partial Least Squares analysis (supervised) with regard to Partial_Least Scores to Interpret the results of such analysis

or you can just apply PCA (unsupervised) to analyse what Components(dims) are Principal in y's variation, as is explaining the most of variance when data is projected onto them. But NB: Pca is used in highly correlated data (to solve the problem of multicollinearity) -- if in scatter plots you see any interactions of features, with PCA you can remove multicollinearity. Besides PCA removes outliers. But with non-normally distributed data - it works poorly

It's all about statistics, but most types of stat.analysis are being done easily in a Pipeline in sklearn, that is not only ML, but 80% of success belongs to input_data handling as Feature Selection algorithms. You can find in sklearn diff. feature_selection & feature_importance algorithms, besides it is stated in documents that Random_Forest is extracting features itself together with making classification. But I don't know exact algorithms behind sklearn (just maximum(for regr)/log(for clf)-likelihood estimations in general), but believe that it is possible to impleement stat. logics with sklearn library, as I already mentioned: needed analysis combining with needed classifier in sklearn's Pipline

having differencies in interpretation in statistical models & predictive models only due to loss-function & metrics used for interpretation... mse just for regression models

P.S. for non-linear dependencies - use SVM (having risk of overfitting if applied to really linear, by nature, dependancies)


If you are looking for detection methods that uncover "shadowed" predictors,

also called latent variables/dimensions


English is not my native language so i may have not understood what's your problem, but if you need to find the best model you can try using a backwards procedure (and eventually adding interations), starting with a model with all covariates. You can then look at both the residuals_vs_predicted values and the qq-plot graphs to check if the model is well describing your phenomenon

  • $\begingroup$ Thanks! I think that what you're suggesting is incrementally adding the most correlated feature. It makes sense, but doesn't help me understand "by how much" feature A is more important than feature B. For example, assume that I have one feature x, and another feature x+<small noise>. Then both are actually useful features, but one is shadowed by the other. I want a method that would also show x+<noise> to be important. $\endgroup$
    – Guy Adini
    Commented Jan 6, 2013 at 16:50
  • $\begingroup$ No, a backward procedure starts with a model with all covariates and then removes a covariate (whose coefficient is not significant) step by step (until you have a model with only significant coefficients, usually). I guess there are more sophisticated ways to achieve the same goal, but i'm just a bachelor student! $\endgroup$
    – Davide
    Commented Jan 7, 2013 at 20:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.