11
$\begingroup$

Suppose a logistic regression model is used to predict whether an online shopper will purchase a product (outcome: purchase), after he clicked a set of online adverts (predictors: Ad1, Ad2, and Ad3).

The outcome is a binary variable: 1 (purchased) or 0 (not purcahsed). The predictors are also binary variables: 1 (clicked) or 0 (not clicked). So all variables are on the same scale.

If the resulting coefficients of Ad1, Ad2, and Ad3 are 0.1, 0.2, and 03, we can conclude that Ad3 is more important than Ad2, and Ad2 is more important than Ad1. Furthermore, since all variables are on the same scale, the standardized and un-standardized coefficients should be same, and we can further conclude that Ad2 is twice important than Ad1 in terms of its influence on the logit (log-odds) level.

But in practice we care more about how to compare and interpret the relative importance of the variables in terms of p(probability of the purchase) level, not the logit(log-odds).

Thus the question is: Is there any approach to quantify the relative importance of these variables in terms of p?

$\endgroup$
  • $\begingroup$ I found this article useful. It describes well six different methods that can be used to define predictor importance from a logistic regression model along with props & cons associated with each method. $\endgroup$ – gchaks Jul 14 '17 at 17:35
5
$\begingroup$

For linear models you can use the absolute value of the t-statistics for each model parameter.

Also, you can use something like a random forrest and get a very nice list of feature importances.

If you are using R check out (http://caret.r-forge.r-project.org/varimp.html), if you are using python check out (http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html#example-ensemble-plot-forest-importances-py)

EDIT:

Since logit has no direct way to do this you can use a ROC curve for each predictor.

For classification, ROC curve analysis is conducted on each predictor. For two class problems, a series of cutoffs is applied to the predictor data to predict the class. The sensitivity and specificity are computed for each cutoff and the ROC curve is computed. The trapezoidal rule is used to compute the area under the ROC curve. This area is used as the measure of variable importance

An example of how this works in R is:

library(caret)
mydata <- data.frame(y = c(1,0,0,0,1,1),
                 x1 = c(1,1,0,1,0,0),
                 x2 = c(1,1,1,0,0,1),
                 x3 = c(1,0,1,1,0,0))

fit <- glm(y~x1+x2+x3,data=mydata,family=binomial())
summary(fit)

varImp(fit, scale = FALSE)
$\endgroup$
  • 1
    $\begingroup$ Thanks for your reply! yes it is easy for linear model and random forest, do you have any idea how to do it in Logistic Regression case? Thanks a lot! $\endgroup$ – xyhzc Jul 9 '14 at 15:26
  • $\begingroup$ See edit above. $\endgroup$ – mike1886 Jul 9 '14 at 15:38
  • $\begingroup$ It seems the question about ratio-level comparisons still hasn't been answered. Even if we know that AUC is, say, .6 using just x1 and .9 using just x2, we can hardly say that x2's importance is therefore 50% greater. Nor, I think, that it's (1 - 10%/40%) = 75% greater. Nor can we do something analogous using just sensitivity or just specificity. I also have doubts about the Wald statistic's applicability here. Most helpful might be explanations of standardized coefficients (see Scott Menard's online book). $\endgroup$ – rolando2 Jul 9 '14 at 16:42
  • $\begingroup$ Thanks rolando2! The variables in this question are all measures in the same metrics, so the standardized and un-standardized coefficients should be the same. Furthermore, although we can use the standardized coefficients to compare the variables on logit (log-odds) level, how can we interpret the variables on P (the probability of online shoppers' purchase in this case)? thanks a lot! $\endgroup$ – xyhzc Jul 10 '14 at 6:59
  • 1
    $\begingroup$ I don't see it answer the question. $\endgroup$ – SmallChess Feb 2 '17 at 14:35
4
$\begingroup$

Since you were specifically asking for an interpretation on the probability scale: In a logistic regression, the estimated probability of success is given by

$\hat{\pi}(\mathbf{x})=\frac{exp(\beta_0+ \mathbf{\beta x})}{1+exp(\beta_0+ \mathbf{\beta x})}$

With $\beta_0$ the intercept, $\mathbf{\beta}$ a coefficient vector and $\mathbf{x}$ your observed values. So if your coefficients are 0.1, 0.2 and 0.3 and supposing no intercept (most likely incorrect, but for easiness), the probability of a purchase for a person who clicked ad 1 only is:

$\frac{exp(0.1)}{1+exp(0.1)}=0.52$

A person who clicked ad 3 only:

$\frac{exp(0.3)}{1+exp(0.3)}=0.57$

However, if the person clicked ad 1 or ad 3 but also ad 2 (if this is a plasubile scenario), the probabilities becomes

$\frac{exp(0.1+0.2)}{1+exp(0.1+0.2)}=0.57$

$\frac{exp(0.3+0.2)}{1+exp(0.3+0.2)}=0.62$

In this case the change in probability is both 0.05, but usually this change is not the same for different combinations of levels. (You can see this easily if you e.g. use the same approach as above but with coefficients 0.1, 1.5, 0.3.) Thus, the importance of a variable on the probability scale is dependent on the observed levels of the other variables. This may make it hard (impossible?) to come up with an absolute, quantitative variable importance measure on the probability scale.

$\endgroup$
  • $\begingroup$ thanks for your explanation! Then do you know is there any indirect method to quantify the relative importance of the predictors? mike1886 mentioned the "ROC curve analysis" in his answer, but is has some issues as mentioned by rolando2 . Thanks a lot! $\endgroup$ – xyhzc Jul 10 '14 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.