2
$\begingroup$

I found in Wikipedia the definition of fraction of variance unexplained in the case of a regressor. Is there a meaningful definition of variance unexplained for a classifier?

$\endgroup$

2 Answers 2

5
$\begingroup$

In case of a 0-1 coded binary outcome, squared error loss on predicted probabilities (a.k.a. Brier score) is most similar to $\mathrm{VAR}_\mathrm{err}$. The variance of a single-trial binomial variable seems most similar to $\mathrm{VAR}_{\mathrm{tot}}$.

Then

$1 - R^2 = \frac{\mathrm{VAR}_{\mathrm{err}}}{\mathrm{VAR}_{\mathrm{tot}}} = \frac{\frac{1}{n}\sum_i (\hat{p}_i - y_i)^2}{\bar{y} (1-\bar{y})}$

where $n$ is the number of observations the measure of fit was computed on, $\hat{p}_i$ is the model-predicted probability of being in class 1 (versus 0) for observation $i$ and $\bar{y}$ is the baserate, or proportion of observations in class 1.

The $R^2$ thus computed is referred to as Efron’s pseudo R-squared. Wikipedia provides some alternative measures, that are not computed based on squared error loss, but on binomial deviance, for example.

$\endgroup$
2
  • 3
    $\begingroup$ While it is totally fair to use an $R^2$-style measure like you give, it lacks the “fraction of variance explained” if the model is nonlinear like most classifiers are (e.g., logistic regression). $\endgroup$
    – Dave
    Nov 22, 2022 at 11:20
  • $\begingroup$ @Dave, agreed! Thanks for links in your answer below, enlightening discussions! $\endgroup$ Nov 22, 2022 at 14:15
2
$\begingroup$

That Wikipedia article is only correct in the case of OLS linear regression. If you estimate the parameters of a linear model another way, such as with regularization, or if you have a nonlinear regression, that interpretation of $R^2$ breaks down.

Interestingly, if you use a linear probability model that you fit via OLS, $R^2$ has the usual interpretation. Linear probability models have their issues, but they do exist, and people do use them in “classification” problems.

In general, however, nonlinear regression models like logistic regressions and neural networks lack the usual interpretation of $R^2$ as the fraction or proportion of variance explained.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.