Why there is square in MSE (mean squared error)?

Question

Please forgive me for such a beginner question, since I'm learning stats . & machine learning.

I'm trying to understand Mean Squared Error.

I understand the "Mean Error", the Mean of Errors between real and predicted values, what worries me is why we take square of errors?

If it's just to keep the values positive then why don't we only take absolute values.

I just want to understand what values does it bring to the actual loss function.

Thanks

Answer 1

MSE has some desirable properties such as easier differentiability (as @user2974951 comments) for further analysis. Differentiability of objective function is in general very important to perform analytical calculations. Taking absolute values is called Mean Absolute Error (MAE in short). It also has applications. It's not like we always prefer MSE or MAE. Another reason, might be penalising large errors more, because if your error is large, its square is much larger. For example, if some error term, $e_i$ is 999, and the other, $e_j$, is $50$; and if we are to choose which term to decrease by an amount of $1$, MAE can choose any of them. But, MSE aims at the larger one since the square decrease is higher.

Answer 2

If $\hat{\theta}$ is an estimator of the parameter $\theta$ then the MSE $\mathbb{E}[(\hat{\theta} - \theta)^2]$ is the sum of the variance of $\hat{\theta}$ and the square bias :

\begin{align*} \mathbb{E}[(\hat{\theta} - \theta)^2] &= \mathbb{E}\big [ \hat{\theta}^2 - 2\hat{\theta}\theta + \theta^2\big ] \\ &= \mathbb{E}[\hat{\theta}^2] -2\theta\mathbb{E}[\hat{\theta}] + \theta^2 \\ &=\mathbb{E}[\hat{\theta}^2] - \mathbb{E}[\hat{\theta}]^2 + \mathbb{E}[\hat{\theta}]^2 - 2\theta\mathbb{E}[\hat{\theta}] + \theta^2 \\ &= \text{Var}(\hat{\theta}) + (\mathbb{E}[\hat{\theta}] - \theta )^2 \\ &= \text{Var}(\hat{\theta}) + \text{Bias}(\hat{\theta})^2 \end{align*}

The MSE is thus made of two important characteristics of an estimator : bias and variance. An estimator may have a small bias but if it has a large variance it's not interesting. On the other hand, an estimator may be very precise, i.e small variance, but if it has a large bias it's also not interesting. The MSE takes both into account.

Moreover, one property of the MSE is that if $\hat{\theta}$ depends on $n$, the size of the sample, then if MSE($\hat{\theta}_n) \to 0$ as $n \to +\infty$ (thus both variance and bias converge to zero) $\hat{\theta}_n$ is consistent, i.e it converges in probability to $\theta$.

Answer 3

I think the some of the answers here aren’t fully answering the question. If we are penalizing the error more wouldn’t squaring it be rather arbitrary? If the mean squared error for one estimator (MSE1) is larger than the mean squared error for another estimator (MSE2) than sqrt(MSE1) > sqrt(MSE2) (proof: https://math.stackexchange.com/questions/1494484/using-proof-by-contradiction-to-show-that-xy-implies-sqrt-x-sqrt-y/1494511). The order is preserved and you are not changing anything by taking the square root and in fact not further penalizing anything.

The mean squared error (MSE) is the “distance” between the true value and the estimated value. The distance you are used to seeing is Euclidean distance in one dimension (i.e. sqrt((difference between two points)^2) ). But, how can we measure the distance between other objects? For example, how can we measure the distance between two functions? At some points of the function, the “y-value” is higher for one function and at other points the “y-value” is higher for the other function. In order to define a distance between two functions, we need a more abstract definition for distance. We will call this abstract distance a metric and we would like it to follow the following properties: 1. the distance between two objects cannot be negative 2. the distance from “A to B” is the same as the distance from “B to A” 3. the distance from “A to C” is less than or equal to the distance from “A to B” plus the distance from “B to C”

So coming back to our example of how to measure the distance between two function, if we define a metric as one function is x distance away from another function by the absolute value of the difference between the maximum y-values of the two functions, then that metric satisfies the three properties. So if g(x) can take values of 1 to 5 for all possible values of x and f(x) can take values 2 to 4 for all values of x, then the distance between g and f is 5–4=1.

Now getting back to your original question, the answer is squaring the difference between the true value and the estimated value satisfies those three properties for distance (so we don’t need to take a square root). It is the same for variance. The variance is the weighted sum of the distances between possible outcomes and the mean. The standard deviation is the square root of the variance. The reason we sometimes use standard deviation as the measure of dispersion is because the variance is in squared units. For example, (5 feet - 1 feet)^2 = 16 feet^2. How can we compare 16 feet^2 with anything that is in just feet? By taking the square root, we can compare 4 feet with other things measured in feet. So to summarize, it doesn’t really matter if you take the square root, it’s still just measuring the distance between two thing. For variance, we want to compare it to other things with the same unit so we use standard deviation. MSE is only being compared to other MSEs so there is no need to take the square root.

Note: some of the things I wrote are not so rigorously shown or stated, but I just wanted to give you the idea of how it works.