6
$\begingroup$

Why can't we simply use ordinary $R^2$ in logistic regression, as we do in linear regression? Domencich and McFadden seem to imply that heteroskedasticity is an issue:

enter image description here

but I don't understand why. In linear regression, homoskedasticity is an assumption (in max. likelihood the variance is typically not considered a parameter to optimise), but why it is an issue for calculating a measure of fit?

Below is a plot from my simulations, for normally distributed classes of different sizes and variances:

McFadden vs. ordinary R^2

McFadden's pseudo-$R^2$ and ordinary $R^2$ seem to be closely related. I'd appreciate if someone could give an illustrative example for a problem arising from heteroskedasticity or any other justification against ordinary $R^2$.

Update:

To avoid confusion: I do not imply using ordinary linear regression on nominal data, and/or calculating squared distances from the straight line. For the purpose of my question I'd perform logistic regression, which would produce a logistic curve. My question is: Why is ordinary $R^2$, as a goodness-of-fitt measure of that curve bad, or worse than McFadden's pseudo-$R^2$.

P.S. (for all those thinking this question is a duplicate):

Please quote the relevant portion of that allegedly existing answer to show how it answers my question. Also, please consider: If something is obvious to you, it need not be obvious to others.

$\endgroup$
8
  • 1
    $\begingroup$ Does this answer your question? McFadden's Pseudo-R2 Interpretation $\endgroup$
    – Nick Cox
    Apr 27, 2020 at 11:13
  • 1
    $\begingroup$ A short answer is that you can use a conventional $R^2$ and some people do. But it's at best descriptive and not directly related to what logistic regression maximizes in fitting. In many circles it would be important to flag that it's non-standard and flak from reviewers would be highly likely if you submitted a paper using it. The bigger fact is that $R^2$ has many limitations even for plain regression and away from that arena no measure does more than capture one facet of what $R^2$ summarizes. $\endgroup$
    – Nick Cox
    Apr 27, 2020 at 11:15
  • $\begingroup$ @Nick Cox: No, sorry, I had already read that link and the references before asking, and they actually led me to the book I quoted. But they don't answer the question "why". $\endgroup$
    – Igor F.
    Apr 27, 2020 at 11:18
  • $\begingroup$ Seems fairly transparent to me as a measure based on log-likelihood. Perhaps we would be better off if it had some different name so that naive readers didn't expect it to behave quite like $R^2$. Someone should have called it McFadden's $M$, or whatever. (Perhaps someone did.) $\endgroup$
    – Nick Cox
    Apr 27, 2020 at 11:24
  • $\begingroup$ @NickCox: OK, that makes sense now. So it has nothing to do with heteroskedasticity. Do you want to post an answer? $\endgroup$
    – Igor F.
    Apr 27, 2020 at 11:31

1 Answer 1

6
$\begingroup$

McFadden's pseudo-$R^2$ is consistent with the log-likelihood model we optimise in logistic regression. The ordinary $R^2$ is consistent with the log-likelihood model for the linear regression.

In linear regression, we maximise the log-likelihood:

$$ - \sum_i (y_i - \beta x_i)^2 $$

Compare this with the definition of $R^2$:

$$ R^2 = 1 - \frac{\sum_i (y_i - \beta x_i)^2}{\sum_i (y_i - \bar y)^2} $$

In the numerator we have the likelihood of our model, and in the denominator the likelihood of the null-model. McFadden's pseudo-$R^2$ is constructed according to the exactly the same schema, just that the log-likelihoods are defined differently for logistic regression. In fact, one can use McFadden's pseudo-$R^2$ for any log-likelihood-based model, just by plugging in the corresponding log-likelihood function. Maybe "generalised $R^2$" would be a more appropriate name.

Homo/heteroskedasticity is not the actual issue here. In linear regression it is implied in the log-likelihood (actually each term in the sum can be thought of as divided by $2 \sigma^2$, but that doesn't matter, since it doesn't change the vector $\beta$). But, in logistic regression, variance doesn't figure at all.*


*Well, it does on a different level, but we have implicitly accepted that as soon as we agreed on using logistic regression in the first place.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.