Can the mean squared error be used for classification? I know the mean squared error formula and how to compute it. When we talk about a regression we can compute the mean squared error. However can we talk about a MSE for a classification problem and how to compute it?
 A: For probability estimates $\hat{\pi}$ you would want to compute not MSE (the log likelihood of a Norma random variable) but instead use likelihood of a Bernoulli random variable
$L=\prod_i \hat{\pi}_i^{y_i} (1-\hat{\pi}_i)^{1-y_i}$
This likelihood is for a binary response, which is assumed to have a Bernoulli distribution.
If you take the log of $L$ and then negate, you get the logistic loss, which is sort of the analog of MSE for when you have a binary response.  In particular, MSE is the negative log likelihood for a continuous response assumed to have a normal distribution.
A: Technically you can, but the MSE function is non-convex for binary classification. Thus, if a binary classification model is trained with MSE Cost function, it is not guaranteed to minimize the Cost function. Also, using MSE as a cost function assumes the Gaussian distribution which is not the case for binary classification. 
A: Many classifiers can predict continuous scores. Often, continuous scores are intermediate results that are only converted to class labels (usually by threshold) as the very last step of the classification. In other cases, e.g. posterior probabilities for the class membership can be calculated (e.g. discriminant analysis, logistic regression).
You can calculate the MSE using these continuous scores rather than the class labels. The advantage of that is that you avoid the loss of information due to the dichotomization.
When the continuous score is a probability, the MSE metric is called Brier's score.
However, there are also classification problems that are rather regression problems in disguise. In my field that could e.g. be classifying cases according to whether the concentration of some substance exceeds a legal limit or not (which is a binary/discriminative two-class problem). Here, MSE is a natural choice due to the underlying regression nature of the task. 
In this paper we explain it as part of a more general framework:
C. Beleites, R. Salzer and V. Sergo:
Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to Grading of Astrocytoma Tissues
Chemom. Intell. Lab. Syst., 122 (2013), 12 - 22. 
How to compute it: if you work in R, one implementation is in package "softclassval", http:/softclassval.r-forge.r-project.org.
A: I don't quite see how... successful classification is a binary variable (correct or not), so it is difficult to see what you would square.
Generally classifications are measured on indicators such as percentage correct, when a classification that has been estimated from a training set, is applied to a testing set that was set aside earlier.
Mean square error can certainly be (and is) calculated for forecasts or predicted values of continuous variables, but I think not for classifications.
