4
$\begingroup$

Let's do some classification and evaluate the prediction quality. The easiest metric to understand is the prediction accuracy, which can be reported as the proportion classified correctly to put the value in the interval $[0,1]$. Is this a reasonable analogue for classification problems of $R^2$ in regression problems?

$\endgroup$
1
  • 1
    $\begingroup$ The shame of all of this is that accuracy is not even a good performance metric (a fact that we now know has a good academic reference). However, I can imagine situations where customers want the accuracy score, and I think it is important to understand what it does and does not mean when such situations arise. "Why use the less accurate model?" discussions can (and should) come later. $\endgroup$
    – Dave
    Feb 15, 2023 at 1:13

2 Answers 2

7
$\begingroup$

A UCLA page gives a number of options for $R^2$ analogues in "classification" problems. The last two options on there ("Count" and "Adjusted Count") relate to the classification accuracy. Indeed, what UCLA calls "Count" on that page is the proportion classified correctly (with some assumptions about how the outputs from a probability model like a logistic regression has its continuous outputs mapped to discrete categories, though classifications being evaluated need not come from a model that produces anything other than a discrete category (such as asking someone and just getting the name of some discrete category)).

It then becomes a matter of what properties of $R^2$ you want your "classification $R^2$" to have.

Even though $R^2$ and the proportion classified correctly exist between zero and one,$^{\dagger}$ the proportion classified correctly has a considerable difference from $R^2$ in that $R^2$ compares the model predictions to a naïve model that always predicts the same value every time, where that value is the pooled/marginal mean of the dependent variable $y$.

$$ R^2 = 1-\dfrac{ \overset{n}{\underset{i=1}{\sum}}\left( y_i -\hat y_i \right)^2 }{ \overset{n}{\underset{i=1}{\sum}}\left( y_i -\bar y \right)^2 } $$

(The notation takes the $y_i$ as observed values, $\hat y_i$ as predicted values, and $\bar y$ as the mean value of all observed $y$.)

To see why this compares your model to a competitor, we could write this a different but equivalent way.

$$ R^2 =1- \dfrac{ \overset{n}{\underset{i=1}{\sum}}\left( y_i -\hat y_i \right)^2 }{ \overset{n}{\underset{i=1}{\sum}}\left( y_i -\hat y^{\prime}_i \right)^2 } $$

Here, the $\hat y_i$ are predictions from your model, while the $\hat y^{\prime}_i$ are the predictions from some competing model, in this case, a model that predicts $\hat y^{\prime}_i = \bar y$ for all $i$.

Anyway, the point is that $R^2$ compares your performance to the performance of some "must beat" model. If you can't do better at predicting conditional means than predicting average(A:A) (to use some Excel terminology) every time, your model probably is not very good.

Proportion classified correctly contains no such comparison to a "must beat" model. To me, that makes accuracy a totally different kind of assessment of performance. Indeed, that accuracy lacks a comparison to a "must beat" model is part of what leads to issues or perceived issues when there is one class with much more representation than the other classes ("class imbalance"). A model with a classification accuracy of $99\%$ sounds like an outstanding model. If one category makes up $99.5\%$ of the cases, however, the model is doing worse than simply predicting the dominant category every time (analogous to predicting $\bar y$ every time in the regression setting).

One thought on a remedy that I have had is to do something similar to accuracy as $R^2$ does to the sum of the squared residuals. $R^2$ puts the sum of the squared residuals in the numerator and the sum of the squared residuals of the "must beat" competing model in the denominator. Doing the same with the misclassification proportion is analogous. In the above example, doing so gives a value of $1-\frac{1 - 0.99}{1-0.995}=1-\frac{0.01}{0.005}=-1$, exposing the model with $99\%$ accuracy as doing worse than naïvely guessing the dominant category every time and not being a particularly good model.

(There are exceptions, such as if you accurately predict important cases, where lower accuracy than the "must beat" model achieves might be acceptable, but that mixes in more decision theory than I believe is appropriate at this time.)

Overall, I would not say that the proportion classified correctly is a reasonable $R^2$-style metric for a classification problem. The only property it has in common with $R^2$ is being bounded between zero and one (including the endpoints), and even that property need not apply to $R^2$ in every setting (e.g., a linear model without an intercept). A reasonable transformation of the proportion classified correctly could be to compare the model error rate (one minus the proportion classified correctly) to the error rate of a model that naïvely predicts the majority class every time.

$$ R^2_{\text{accuracy}} = 1 - \dfrac{ \text{Error rate of the model under consideration} }{ \text{Error rate of a model that naïvely predicts the majority class every time} } $$

This makes sense to me. If your "must beat" model achieves an error rate of $20\%$ and you build a model with an error rate of $10\%$, this calculation gives the reasonable answer that you have halved the error rate. If, instead, your model achieves an error rate of $1\%$, you get a $95\%$ reduction in the error rate. If, finally, your model achieves an error rate of $60\%$, you get a number that reports a tripling of the error rate (the value of $-2$ corresponds to a $200\%$ increase in the error rate, so a tripling).

(If someone wants to report the values as $\text{fraction}-1$ instead of $1-\text{fraction}$ in order to have error rate improvements be negative and error rate increases be positive, that could make a lot of sense, too. My big concern is to consider a comparison to a "must beat" model. Then the values above would be $-0.5$ ($50\%$ reduction in error rate), $-0.95$ ($95\%$ reduction in error rate), and $+2$ ($200\%$ increase in error rate).)

$^{\dagger}$Proportion classified correctly is restricted to $[0,1]$, but for $R^2$, it depends on what calculation you do (not everyone even agrees on the definition of $R^2$ outside of simple situations), the data on which you perform the calculation, and the model to which the data are fit. That the usual $R^2$ need not be bounded below yet accuracy is bounded below by zero could be another strike against accuracy as being an classification analogue of $R^2$. The UCLA page even seems to concede this point.

Count R-Squared does not approach goodness of fit in a way comparable to any OLS approach.

EDIT

The adjusted count given by UCLA turns out to be equivalent to what I call $R^2_{accuracy}$ here.

How to interpret the UCLA "adjusted count" logistic regression pseudo $R^2?$

$\endgroup$
3
  • 1
    $\begingroup$ Very nice, +1. If we use probabilistic class membership predictions $0\leq \hat{y}\leq 1$, then calculate $R^2$ straightforwardly, then it's the ratio between the Brier scores of the focal model and an "overall average" model. $\endgroup$ Feb 15, 2023 at 7:21
  • $\begingroup$ @StephanKolassa The UCLA page mentions that one as Efron’s $R^2$. // It turns out that my $R^2_{accuracy}$ was hiding in plain sight this whole time as UCLA’s adjusted count! $\endgroup$
    – Dave
    Feb 18, 2023 at 15:21
  • 1
    $\begingroup$ +1. I'm very sure you can derive this identity starting from likelihood ratios of Dirac deltas, one for the model being investigated, another for a constant model. In the case of the Dirac likelihood, the constant parameter that minimizes the loss is the majority class. $\endgroup$
    – Firebug
    Sep 7, 2023 at 11:46
0
$\begingroup$

The direct analogue of R^2 for classification models are pseudo-rsquared values https://en.m.wikipedia.org/wiki/Pseudo-R-squared and it is related to the concept of deviance. Assuming a homoscedastic gaussian log likelihood will give you the normal R2 in the case of normal regression models.

$\endgroup$
1
  • 1
    $\begingroup$ It’s more complicated than that. You don’t need any explicit reference to Gaussian likelihood to derive the usual $R^2$ formula, so it’s not obvious that a generalization for logistic regression should refer to binomial likelihood or deviance. That UCLA page in my answer gives multiple reasonable ways to approach an $R^2$-like value in more generality, not all of which use likelihood or deviance. $\endgroup$
    – Dave
    Sep 7, 2023 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.