variance unexplained for binary classifier I found in Wikipedia the definition of fraction of variance unexplained in the case of a regressor. Is there a meaningful definition of variance unexplained for a classifier?
 A: In case of a 0-1 coded binary outcome, squared error loss on predicted probabilities (a.k.a. Brier score) is most similar to $\mathrm{VAR}_\mathrm{err}$. The variance of a single-trial binomial variable seems most similar to $\mathrm{VAR}_{\mathrm{tot}}$.
Then
$1 - R^2 = \frac{\mathrm{VAR}_{\mathrm{err}}}{\mathrm{VAR}_{\mathrm{tot}}} = \frac{\frac{1}{n}\sum_i (\hat{p}_i - y_i)^2}{\bar{y} (1-\bar{y})}$
where $n$ is the number of observations the measure of fit was computed on, $\hat{p}_i$ is the model-predicted probability of being in class 1 (versus 0) for observation $i$ and $\bar{y}$ is the baserate, or proportion of observations in class 1.
The $R^2$ thus computed is referred to as Efron’s pseudo R-squared. Wikipedia provides some alternative measures, that are not computed based on squared error loss, but on binomial deviance, for example.
A: That Wikipedia article is only correct in the case of OLS linear regression. If you estimate the parameters of a linear model another way, such as with regularization, or if you have a nonlinear regression, that interpretation of $R^2$ breaks down.
Interestingly, if you use a linear probability model that you fit via OLS, $R^2$ has the usual interpretation. Linear probability models have their issues, but they do exist, and people do use them in “classification” problems.
In general, however, nonlinear regression models like logistic regressions and neural networks lack the usual interpretation of $R^2$ as the fraction or proportion of variance explained.
