R squared in logistic regression adjusted for number of predictors

Question

For OLS we have an adjusted R squared which adjusts for the number of predictors included in the model.

For logistic regression there are some R squared analogues (Tjur’s R squared, McFadden’s R squared, Cox-Snell’s R squared and Nagelkerke’s R squared). But is there a R squared measure for logistic regression that adjusts for the number of perdictors included in the model?

This post about R squared for logistic regression does not answer my question because it has no info on an R squared that is adjusted for the number of predictors in the model. The "adjustment" being mentioned there refers to the scale of the measure with upper limit being 1. This is not what my quetion is about.

Answer 1

I discuss this here and especially here. My favorite general-purpose adjusted $R^2$ is

$R^{2}_{p, m} = 1 - \exp((\text{LR} - p) / m)$

where $p$ is the number of parameters in the predictors whose added value you are quantifying, and $m$ is the effective sample size. For binary logistic regression $m=3np(1-p)$ where $p$ is the proportion of $Y=1$. For Cox PH models $m$ is the number of events. LR is the likelihood ratio $\chi^2$ added by the predictors. Note that the R rms package anova function will give you all the LR statistics.

Also see relative explained variation.

Answer 2

A resource I have found useful is the UCLA page on pseudo $R^2$ for logistic regression. Note that their last measure is equivalent to the accuracy transformation I show here, which I prove here.

That UCLA page gives an "adjusted" $R^2$ that penalized the inclusion of additional features of:

$$ R^2_{adj} = 1 - \dfrac{ \log\left(\text{L}_{\text{Model}}\right) - K }{ \log \left(\text{L}_{\text{Null}}\right) } $$

$\log\left(\text{L}_{\text{Model}}\right)$ is the log-likelihood of the model. $\log\left(\text{L}_{\text{Null}}\right)$ is the log-likelihood of an intercept-only model. $K$ is the parameter count, but the page does not seem to specify if that includes the intercept.

McFadden’s adjusted mirrors the adjusted R-squared in OLS by penalizing a model for including too many predictors. If the predictors in the model are effective, then the penalty will be small relative to the added information of the predictors. However, if a model contains predictors that do not add sufficiently to the model, then the penalty becomes noticeable and the adjusted R-squared can decrease with the addition of a predictor, even if the R-squared increases slightly. Note that negative McFadden’s adjusted R-squared are possible.

The fact that this adjusted value can decrease upon adding a new feature is an appealing feature retained from the usual adjusted $R^2$.

But Harrell gives an alternative calculation that also penalizes the inclusion of fairly unhelpful features.

But why subtract the feature count? Why not subtract two times the feature count or a logarithm of the feature count?

For the usual adjusted $R^2$, there is an interpretation related to the ratio of two unbiased variance estimates.

In a logistic regression, such an interpretation presents problems, because there is not a sense in which the outcome has a constant variance unless all predicted probabilities are the same, so there goes the idea of one "error variance" value that could go into the numerator, let alone an unbiased estimator of something that does not exist.

Finally, there is not even an unambiguous $R^2$ calculation for logistic regression, as that UCLA page shows. Calculations can reasonably use square loss or binomial log-likelihood, even a few other notions.

With all of this in mind, there is not a single "adjusted R-squared" for logistic regressions. It comes down to what you want to know from your calculated value. I find this worth keeping in mind when you decide on the statistic or statistics you calculate.