13
$\begingroup$

I know the mean squared error formula and how to compute it. When we talk about a regression we can compute the mean squared error. However can we talk about a MSE for a classification problem and how to compute it?

$\endgroup$
12
$\begingroup$

Many classifiers can predict continuous scores. Often, continuous scores are intermediate results that are only converted to class labels (usually by threshold) as the very last step of the classification. In other cases, e.g. posterior probabilities for the class membership can be calculated (e.g. discriminant analysis, logistic regression). You can calculate the MSE using these continuous scores rather than the class labels. The advantage of that is that you avoid the loss of information due to the dichotomization.
When the continuous score is a probability, the MSE metric is called Brier's score.

However, there are also classification problems that are rather regression problems in disguise. In my field that could e.g. be classifying cases according to whether the concentration of some substance exceeds a legal limit or not (which is a binary/discriminative two-class problem). Here, MSE is a natural choice due to the underlying regression nature of the task.

In this paper we explain it as part of a more general framework: C. Beleites, R. Salzer and V. Sergo:
Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to Grading of Astrocytoma Tissues
Chemom. Intell. Lab. Syst., 122 (2013), 12 - 22.

How to compute it: if you work in R, one implementation is in package "softclassval", http:/softclassval.r-forge.r-project.org.

$\endgroup$
0
$\begingroup$

I don't quite see how... successful classification is a binary variable (correct or not), so it is difficult to see what you would square.

Generally classifications are measured on indicators such as percentage correct, when a classification that has been estimated from a training set, is applied to a testing set that was set aside earlier.

Mean square error can certainly be (and is) calculated for forecasts or predicted values of continuous variables, but I think not for classifications.

$\endgroup$
0
$\begingroup$

For probability estimates $\hat{\pi}$ you would want to compute not MSE but instead the likelihood:

$L=\prod_i \hat{\pi}_i^{y_i} (1-\hat{\pi}_i)^{1-y_i}$

This likelihood is for a binary response, which is assumed to have a Bernoulli distribution.

If you take the log of $L$ and then negate, you get the logistic loss, which is sort of the analog of MSE for when you have a binary response. In particular, MSE is the negative log likelihood for a continuous response assumed to have a normal distribution.

$\endgroup$
0
$\begingroup$

Technically you can, but the MSE function is non-convex for binary classification. Thus, if a binary classification model is trained with MSE Cost function, it is not guaranteed to minimize the Cost function. Also, using MSE as a cost function assumes the Gaussian distribution which is not the case for binary classification.

$\endgroup$
  • 1
    $\begingroup$ Why would MSE assume Gaussian distribution? (As opposed to, say, least squares regression uses MSE as loss, and we can show that it is optimal for regression problems with normally distributed residuals) $\endgroup$ – cbeleites supports Monica Feb 14 at 12:12
  • $\begingroup$ It is not optimal for binary classification but optimal for regression. The question was for binary. $\endgroup$ – Mostafa Nakhaei Feb 18 at 15:59
  • $\begingroup$ The question doesn't say binary classification. It doesn't even say discriminative classification. And it doesn't ask about optimality (for which you'd need to be still more specific about the situation even than saying binary or discriminative with 2 classes), just whether MSE can be used. Also, Brier's score is a strictly proper scoring rule for forecasting, so a more detailed explanation of the non-optimality would certainly be helpful (and possibly very illuminating as to when this non-optimality applies). $\endgroup$ – cbeleites supports Monica Feb 19 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.