How to quantify the Relative Variable Importance in Logistic Regression in terms of p?

Question

Suppose a logistic regression model is used to predict whether an online shopper will purchase a product (outcome: purchase), after he clicked a set of online adverts (predictors: Ad1, Ad2, and Ad3).

The outcome is a binary variable: 1 (purchased) or 0 (not purcahsed). The predictors are also binary variables: 1 (clicked) or 0 (not clicked). So all variables are on the same scale.

If the resulting coefficients of Ad1, Ad2, and Ad3 are 0.1, 0.2, and 03, we can conclude that Ad3 is more important than Ad2, and Ad2 is more important than Ad1. Furthermore, since all variables are on the same scale, the standardized and un-standardized coefficients should be same, and we can further conclude that Ad2 is twice important than Ad1 in terms of its influence on the logit (log-odds) level.

But in practice we care more about how to compare and interpret the relative importance of the variables in terms of p(probability of the purchase) level, not the logit(log-odds).

Thus the question is: Is there any approach to quantify the relative importance of these variables in terms of p?

Answer 1

For linear models you can use the absolute value of the t-statistics for each model parameter.

Also, you can use something like a random forrest and get a very nice list of feature importances.

If you are using R check out (http://caret.r-forge.r-project.org/varimp.html), if you are using python check out (http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html#example-ensemble-plot-forest-importances-py)

EDIT:

Since logit has no direct way to do this you can use a ROC curve for each predictor.

For classification, ROC curve analysis is conducted on each predictor. For two class problems, a series of cutoffs is applied to the predictor data to predict the class. The sensitivity and specificity are computed for each cutoff and the ROC curve is computed. The trapezoidal rule is used to compute the area under the ROC curve. This area is used as the measure of variable importance

An example of how this works in R is:

library(caret)
mydata <- data.frame(y = c(1,0,0,0,1,1),
                 x1 = c(1,1,0,1,0,0),
                 x2 = c(1,1,1,0,0,1),
                 x3 = c(1,0,1,1,0,0))

fit <- glm(y~x1+x2+x3,data=mydata,family=binomial())
summary(fit)

varImp(fit, scale = FALSE)

Answer 2

Since you were specifically asking for an interpretation on the probability scale: In a logistic regression, the estimated probability of success is given by

$\hat{\pi}(\mathbf{x})=\frac{exp(\beta_0+ \mathbf{\beta x})}{1+exp(\beta_0+ \mathbf{\beta x})}$

With $\beta_0$ the intercept, $\mathbf{\beta}$ a coefficient vector and $\mathbf{x}$ your observed values. So if your coefficients are 0.1, 0.2 and 0.3 and supposing no intercept (most likely incorrect, but for easiness), the probability of a purchase for a person who clicked ad 1 only is:

$\frac{exp(0.1)}{1+exp(0.1)}=0.52$

A person who clicked ad 3 only:

$\frac{exp(0.3)}{1+exp(0.3)}=0.57$

However, if the person clicked ad 1 or ad 3 but also ad 2 (if this is a plasubile scenario), the probabilities becomes

$\frac{exp(0.1+0.2)}{1+exp(0.1+0.2)}=0.57$

$\frac{exp(0.3+0.2)}{1+exp(0.3+0.2)}=0.62$

In this case the change in probability is both 0.05, but usually this change is not the same for different combinations of levels. (You can see this easily if you e.g. use the same approach as above but with coefficients 0.1, 1.5, 0.3.) Thus, the importance of a variable on the probability scale is dependent on the observed levels of the other variables. This may make it hard (impossible?) to come up with an absolute, quantitative variable importance measure on the probability scale.