5
$\begingroup$

For OLS we have an adjusted R squared which adjusts for the number of predictors included in the model.

For logistic regression there are some R squared analogues (Tjur’s R squared, McFadden’s R squared, Cox-Snell’s R squared and Nagelkerke’s R squared). But is there a R squared measure for logistic regression that adjusts for the number of perdictors included in the model?

This post about R squared for logistic regression does not answer my question because it has no info on an R squared that is adjusted for the number of predictors in the model. The "adjustment" being mentioned there refers to the scale of the measure with upper limit being 1. This is not what my quetion is about.

$\endgroup$
2
  • 1
    $\begingroup$ I would not call any of these "equivalents" to $R^2$ There is no exact equivalent. "Analogues" might be a better term. $\endgroup$
    – Peter Flom
    Commented May 28 at 19:14
  • $\begingroup$ @PeterFlom will change it, thanks $\endgroup$
    – LulY
    Commented May 28 at 19:15

2 Answers 2

10
$\begingroup$

I discuss this here and especially here. My favorite general-purpose adjusted $R^2$ is

$R^{2}_{p, m} = 1 - \exp((\text{LR} - p) / m)$

where $p$ is the number of parameters in the predictors whose added value you are quantifying, and $m$ is the effective sample size. For binary logistic regression $m=3np(1-p)$ where $p$ is the proportion of $Y=1$. For Cox PH models $m$ is the number of events. LR is the likelihood ratio $\chi^2$ added by the predictors. Note that the R rms package anova function will give you all the LR statistics.

Also see relative explained variation.

$\endgroup$
1
  • 4
    $\begingroup$ The book REGRESSION MODELING STRATEGIES is pure gold!!! $\endgroup$
    – LulY
    Commented May 28 at 20:09
4
$\begingroup$

A resource I have found useful is the UCLA page on pseudo $R^2$ for logistic regression. Note that their last measure is equivalent to the accuracy transformation I show here, which I prove here.

That UCLA page gives an "adjusted" $R^2$ that penalized the inclusion of additional features of:

$$ R^2_{adj} = 1 - \dfrac{ \log\left(\text{L}_{\text{Model}}\right) - K }{ \log \left(\text{L}_{\text{Null}}\right) } $$

$\log\left(\text{L}_{\text{Model}}\right)$ is the log-likelihood of the model. $\log\left(\text{L}_{\text{Null}}\right)$ is the log-likelihood of an intercept-only model. $K$ is the parameter count, but the page does not seem to specify if that includes the intercept.

McFadden’s adjusted mirrors the adjusted R-squared in OLS by penalizing a model for including too many predictors. If the predictors in the model are effective, then the penalty will be small relative to the added information of the predictors. However, if a model contains predictors that do not add sufficiently to the model, then the penalty becomes noticeable and the adjusted R-squared can decrease with the addition of a predictor, even if the R-squared increases slightly. Note that negative McFadden’s adjusted R-squared are possible.

The fact that this adjusted value can decrease upon adding a new feature is an appealing feature retained from the usual adjusted $R^2$.

But Harrell gives an alternative calculation that also penalizes the inclusion of fairly unhelpful features.

But why subtract the feature count? Why not subtract two times the feature count of a logarithm of the feature count?

For the usual adjusted $R^2$, there is an interpretation related to the ratio of two unbiased variance estimates.

In a logistic regression, such an interpretation presents problems, because there is not a sense in which the outcome has a constant variance unless all predicted probabilities are the same, so there goes the idea of one "error variance" value that could go into the numerator, let alone an unbiased estimator of something that does not exist.

Finally, there is not even an unambiguous $R^2$ calculation for logistic regression, as that UCLA page shows. Calculations can reasonably use square loss or binomial log-likelihood, even a few other notions.

With all of this in mind, there is not a single "adjusted R-squared" for logistic regressions. It comes down to what you want to know from your calculated value. I find this worth keeping in mind when you decide on the statistic or statistics you calculate.

$\endgroup$
3
  • 1
    $\begingroup$ Great information. Apparently unhelpful features must be accounted for otherwise $R^2$ is biased high. The measure I propose is a close approximation of the standard $R^{2}_\text{adj}$ in linear models and could be called an “in-sample unbiased estimator”. The McFadden index is much more severe than the standard adjustment, being an “out of sample debíaser” that is computed from AIC. So a key question is do you want an index to give you an overfitting-corrected $R^2$ or do you want it to tell you whether you get better out-of-sample predictions? $\endgroup$ Commented May 29 at 11:27
  • $\begingroup$ "Calculations can reasonably use square loss or binomial log-likelihood, even a few other notions." - sounds like the decision between the Brier score (=square loss) and the log loss, no? Which would make a lot of sense, since we are essentially optimizing a proper scoring rule. $\endgroup$ Commented May 29 at 14:09
  • $\begingroup$ @StephanKolassa Agreed, since those pseudo $R^2$ values are just transformations of Brier score and log loss. $\endgroup$
    – Dave
    Commented May 29 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.