Computing the variance explained by a predictor variable in logistic regression

Question

I'm keen to know how to compute the variance explained by a particular predictor variable in the model (say component specific R squared). I went through Calculate variance explained by each predictor in multiple regression using R but I'm not clear about the explanation. For simplicity, let me give an example and raise the question in terms of the example.

> y <- rbinom(100, 1, 0.2)
> x1 <- rnorm(100)
> x2 <- rnorm(100)
> m1 <- glm(y~x1+x2, family = binomial(link="logit"))
> summary(m1)

Call:
glm(formula = y ~ x1 + x2, family = binomial(link = "logit"))

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.8905  -0.6764  -0.5979  -0.4634   2.1806  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  -1.5240     0.2713  -5.617 1.94e-08 ***
x1            0.2023     0.2431   0.832    0.405    
x2           -0.3284     0.2451  -1.340    0.180    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 97.245  on 99  degrees of freedom
Residual deviance: 94.624  on 97  degrees of freedom
AIC: 100.62

Number of Fisher Scoring iterations: 4

Simply, I want to quantify the contribution (as a proportion of variance explained) of x2, to the model. What I know is, in the event where y is continuous, x2 is scaled, then the square of the regression coefficient of x2 is a close enough approximation to the proportion of variance explained by x2. In the event of binary logistic, I know that R squared of x2 is not the square of -0.3284. If not that, then what? I need to know how to compute this quantity in logistic regression situation.

Answer 1

There is no exact equivalent of $R^2$ for logistic regression; there is no statistic that will tell you what you are asking for.

There are a number of pseudo-$R^2$ measures. Paul Allison has a very good summary of these in Statistical Horizons (What's the Best R-Squared for Logistic Regression?) But he does not come to strong conclusions. I'm not sure if he has updated this article. There are lots of other articles about this, just Google "pseudo r squared logistic regression".

Answer 2

The gold standard measure of association in the frequentist world is the likelihood ratio (LR) $\chi^2$ statistic. Compute LR for the overall model and the partial LR $\chi^2$ for each predictor (this will be a "chunk" test when a predictor has more than one term in the model). The ratio of the partial LR to the full model LR is a fine measure of relative predictive information for the predictor. See this for an example where these ratios are computed automatically by computing all relevant chunk tests automatically. The example also presents a second way to quantify added value of a predictor that will be in an upcoming release of the R rms package. More ways are here.

Answer 3

My suggestion is to think hard about what you really want to know. Explained variance gets funky once you move from the relatively straightforward situations of OLS linear regression. If a customer is demanding this, press them on what they want to know, or go with your judgment of what they want to know or should want to know.

The usual measure of the proportion of variance explained in a regression is the $R^2$ value: $R^2 = 1-\dfrac{\sum_i(y_i - \hat{y_i})^2}{\sum_i(y_i - \bar{y})^2}$. Here, I derive how this value corresponds to the proportion of variance explained in OLS linear regression. The key point there is that, once you leave the world of least squares linear regression, there is more to the total sum of squares than the regression and residual sum of squares. It is not at all obvious to me what to do with that third component of the total sum of squares, which is why I am comfortable using $R^2$ as a transformation of square loss that might make more sense than just the sum or mean of the squared residuals, but I stop short of saying that the above equation corresponds with the proportion of variance explained.

You're looking at individual features, so some kind of partial $R^2$, but a similar idea exists. With the differences your model has from linear regression, the usual $R^2$ does not necessarily correspond to the proportion of variance explained.

I see three reasonable approaches.

1. Just do the usual calculation for a partial $R^2$. Nothing stops you from using square loss to evaluate a logistic regression. This is the Brier score, and the pseudo $R^2$ based on Brier score is Efron's pseudo $R^2$, as is discussed on this UCLA page.

2. Drag along that cross term in the linked derivation (or the analogous derivation for partial $R^2$). This, arguably, relates more to the proportion of variance explained than just assuming that term to be zero like it is in OLS linear regression.

3. Calculate partial pseudo $R^2$ based on the binomial likelihood, as discussed in another answer. This relates to the McFadden pseudo $R^2$.

I think I would go with the third option while noting that this is more accurately described as assessing the proportion of deviance explained instead of variance explained. However, binomial deviance is somewhat more natural to use than square loss (Gaussian deviance) for a binomial outcome, so I think such an assessment is really getting at the key aspects of the inquiry into which features have the most influence over the outcome.