3
$\begingroup$

I am studying linear regression and I solved some problems analytically. For that I used the normal and intuitive sum of squared error function.

Looking at this function, it makes all sense why it looks the way it does. I mean we have the square because we can have positive and negative errors and we do not want that they eliminate each other and we use the square, because we get exactly one minima, which is a solvable minimization problem. We can use also gradient descent.

Looking at other cost functions, like the mean squared error, I do not understand why it is a good idea to calculate the mean of errors, I mean what motivates us to do this?

I mean it is just an optimization problem, we want the best fit, so I would never get the idea to use the mean of errors and why it makes our optimization problem better.

Is it just to work with a smaller error values, so when using gradient descent it converts faster?

would be great if someone could help me with the intuition and the mathematical motivation. thanks in advance!

$\endgroup$
2
  • $\begingroup$ What would you propose as an alternative to MSE, the sum of the squared errors? $\endgroup$
    – Dave
    Aug 4, 2021 at 17:49
  • $\begingroup$ There is a subtle and practical reason why MSE is preferred over the sum of squared-errors cost function, and it has to do with iterative descent methods. More details in my answer below. $\endgroup$
    – mhdadk
    Aug 4, 2021 at 18:17

3 Answers 3

4
$\begingroup$

It is a fact from calculus that some function $f(x)$ and $cf(x)$ have the same argmin ($x$ that minimizes the function), unless $c=0$. It follows that the following all have the same argmin, thus give the same parameter estimates.

$$ \sum(y_i - \hat y_i)^2\\ \dfrac{\sum(y_i - \hat y_i)^2}{n}\\ \dfrac{\sum(y_i - \hat y_i)^2}{n-p}\\ \dfrac{\sum(y_i - \hat y_i)^2}{8}\\ $$

The first is the usual sum of squared errors. The next two are variants of mean squared error (the $n-p$ denominator has to do with getting an unbiased estimate of the variance of the error term). I made up the final one.

However, all of these give the same parameter estimates (barring numerical issues coming from doing math on a computer).

Mean squared error has the advantage of giving some sense of by how much predictions and true values differ (though this is not perfect, since it is not absolute error), and it has a relationship to the variance of the error term. Further, you do not make the value arbitrarily large by having many observations.

$\endgroup$
7
  • $\begingroup$ I once answered a related question on the Data Science Stack. $\endgroup$
    – Dave
    Aug 4, 2021 at 18:03
  • $\begingroup$ Would it be an adequate paraphrase that we take the mean (1) to avoid the limitations in how computers store/do math with (large) numbers and (2) for more human-interpretability of the loss value? $\endgroup$ Aug 4, 2021 at 18:25
  • $\begingroup$ sorry, but I am still not getting it :( we have an optimization problem that we want to solve, so we want to calculate the coefficients. Do you also agree with me that we can solve this optimization problem by solving min(sum of the squared errors)? I hope the answer is yes :) What I am still not getting is why taking the mean of errors and what I gain from it. I mean SSE does the job, why adding this extra step of calculating the mean? Is the line that I get with the sum of squared errors function different from the line that I get when using Mean Squared Error function? $\endgroup$
    – John Smith
    Aug 4, 2021 at 18:26
  • 1
    $\begingroup$ @AryaMcCarthy My comment about large values was not about computers storing large numbers, but that might be an advantage. // John Smith, you have an explanation from another member about why gradient descent might prefer MSE. $\endgroup$
    – Dave
    Aug 4, 2021 at 18:31
  • 2
    $\begingroup$ @JohnSmith Yes, we agree that the argmin of the cost function (the optimal parameters; the line you get) is unchanged. Shift your attention to the value of the cost function. That’s what taking the mean is for. It (1) keeps the value in a range expressible+usable by computers and (2) makes a number you can interpret+compare across samples, regardless of the size of the sample. (The SSE depends on how many terms you’re summing. Consider the case of millions/billions of data points.) $\endgroup$ Aug 4, 2021 at 18:33
3
$\begingroup$

Let the mean squared-error (MSE) cost function be $$ \mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^N (y_i - f(x_i,\theta))^2 $$ where $x_i$ represents the $i^{\text{th}}$ input, $y_i$ represents the $i^{\text{th}}$ target, and $\theta$ represents the parameters. The goal is to find the $\theta$ that minimizes the MSE cost function.

The number $\frac{1}{N}$ does not change which $\theta$ minimizes the MSE cost function, since it just scales the cost. However, as you have guessed already, it plays a practical role when performing gradient descent, or any other iterative procedure involving gradients and/or Hessians.

The gradient descent update rule is $$ \theta_{k+1} \leftarrow \theta_k - \alpha \cdot \nabla_{\theta} \mathcal{L}(\theta) $$ where $\nabla_{\theta} \mathcal{L}(\theta)$ is the gradient of the MSE cost function with respect to $\theta$. In any descent procedure, you have to choose two things: the direction that you want to descend, and how much you want to descend in that direction. In the gradient descent update rule above, we have already chosen the direction, it is $\nabla_{\theta} \mathcal{L}(\theta)$. However, we have not yet chosen how much we want to descend in that direction. This is done by choosing an appropriate $\alpha$, also known as the "step size".

How do we pick an appropriate $\alpha$? This is what line search is for. In line search, specifically backtracking line search, we want to choose the $\alpha$ that satisfies some specific conditions. It turns out that if the gradient $\nabla_{\theta} \mathcal{L}(\theta)$ is large, such as when you are far away from the origin on a parabola, it takes more time/computation to find an $\alpha$ value that satisfies these conditions. Therefore, we ideally want the values of $\nabla_{\theta} \mathcal{L}(\theta)$ to be small.

The MSE cost function inherently keeps $\nabla_{\theta} \mathcal{L}(\theta)$ small using $\frac{1}{N}$. To see this, suppose that we instead use the sum of squared-errors (SSE) cost function $$ \tilde{\mathcal{L}}(\theta) = \sum_{i=1}^N (y_i - f(x_i,\theta))^2 $$ and so the gradient descent update rule becomes $$ \theta_{k+1} \leftarrow \theta_k - \alpha \cdot \nabla_{\theta} \tilde{\mathcal{L}}(\theta) $$ To keep $\nabla_{\theta} \tilde{\mathcal{L}}(\theta)$ small, let us multiply it by $\frac{1}{N}$ such that $$ \theta_{k+1} \leftarrow \theta_k - \alpha \cdot \frac{1}{N}\nabla_{\theta} \tilde{\mathcal{L}}(\theta) $$ Since the gradient operator is linear, this becomes $$ \theta_{k+1} \leftarrow \theta_k - \alpha \cdot \nabla_{\theta} \frac{1}{N} \tilde{\mathcal{L}}(\theta) $$ But since $$ \frac{1}{N} \tilde{\mathcal{L}}(\theta) = \mathcal{L}(\theta) $$ In other words, $\frac{1}{N}$ times the SSE cost function is the MSE cost function. So, the gradient descent update rule with small gradients is $$ \theta_{k+1} \leftarrow \theta_k - \alpha \cdot \nabla_{\theta}\mathcal{L}(\theta) $$ In fact, notice that as $N$ grows, the SSE cost function grows too, and so does its gradient. Conversely, while the MSE cost function may also grow in value as $N$ increases, it does not grow as fast as the SSE cost function, and so its gradients are also growing slowly.

$\endgroup$
5
  • 2
    $\begingroup$ If you're using gradient descent for linear regression, you're doing it wrong $\endgroup$
    – Hong Ooi
    Aug 5, 2021 at 2:43
  • 2
    $\begingroup$ @HongOoi, are you sure this applies in all situations, all sizes and dimensions of datasets etc.? $\endgroup$ Aug 5, 2021 at 7:47
  • 2
    $\begingroup$ @HongOoi Why? Have you ever worked with very large data sets? See here: stats.stackexchange.com/a/278794/176202 $\endgroup$ Aug 5, 2021 at 8:44
  • $\begingroup$ I feel this is a more insightful answer from the perspective that how this question is raised. $\endgroup$
    – xappppp
    Feb 11 at 14:50
  • $\begingroup$ This answer assumes gradient descent as an optimizer. But.. MSE was used to with OLS long before gradient was a popular choice.. $\endgroup$ Apr 24 at 19:01
1
$\begingroup$

After a lot of discussions and having the luck to talk with some phds, I want to share this answer that I really like.

Let's say we have a model that we trained. How can we say if that is a good or bad model?

Well, to answer this question we need something to validate our model. For that we are using something that we call a metric. In math we know many metrics, which are able to measure stuff.

Now there are different ways to measure stuff. In our case we want to measure the errors to see how good our trained model is.

The best would be if our metric can give us something that is easy to reason about and interpret.

The metric that we pick, should be something that has some meaning in human terms, so with other words it should be easy to reason about and to compare with other values and to interpret.

Because of these requirement, one idea is to use a metric, that square sums all errors up and gives us the mean of this sum. The root of this mean is a representative that tells us that if we do not have outliers, that the values of our model, should be close to this mean value.

To understand that let's say that our trained model gets 5 input values and outputs [3, 3, 4, 4, 2].

Is our model good? Are these values a good prediction?

Well to answer this question, we use the metric that we used to train/optimize hour model.

So we use the mean of squared errors (MSE) and give it the same 5 values our model got. Let's say we got 16. Now we calculate the square root and we get 4.

This 4 is a representative value that is telling us, that the best approximation for hour predictions should be close to 4. This shows how we used the MSE to compare/evaluate/test our model.

When we train a regression model, we always can evaluate and test it with a metric, with other words we can see how good the model is.

Many ML frameworks use the same metric for training and test, because we can reuse the same implementation for training and testing/evaluation.

Looking at it in a mathematical point of view, the squared sum of errors (SSE) and the MSE have the same same minimization value.

If we take our cost function, or any function in general, that we want to optimize, which means to find a minimum or maximum, we can multiply any constant to it, the solutions will never change.

We remember what we learned in our calculus classes. We take the derivative of the function, set it to zero and then solve for the parameters.

For Example:

Let f'(x) = x + 1. Now to find the extrem point, we have to set f'(x) to zero, so we get x + 1 = 0.

x + 1 = 0 has the same solution like a(x + 1) = a 0, so it does not matter what factor we use with our function we want to optimize.

This means min(SSE) = min(MSE), because the only difference between the SSE and the MSE is a factor (like the a in the line above) that does not influence the solution of the minimization.

This shows that the SSE can also be used to train our model, but it's bad to evaluate the model, because the meaning is hidden and hard to interpret, which is why we use the MSE.

So the reason why the literature prefers the MSE for training over the SSE is just a historic convention. The convention says to use the metric for training, that you want to use to test your model with. Which is a nice and consistent convention.

$\endgroup$
6
  • $\begingroup$ What about $4$ being the best approximation? I do not follow your rationale there. $\endgroup$
    – Dave
    Aug 6, 2021 at 20:05
  • $\begingroup$ in this example, 4 is just the mean and you can show that the arithmetic mean is just the best approximation of n numbers (as long you do not have outliers) so in this example we just have a perfect model with perfect data and our metric also agrees with that :) $\endgroup$
    – John Smith
    Aug 6, 2021 at 20:33
  • $\begingroup$ Wouldn’t the perfect model perfectly predict each of the distinct numbers? $\endgroup$
    – Dave
    Aug 6, 2021 at 20:35
  • $\begingroup$ that is correct, I mean perfect in terms of "it works well with the metric together" $\endgroup$
    – John Smith
    Aug 6, 2021 at 21:03
  • $\begingroup$ My concern is that you’re mixing up the desire to use square loss (SSE, MSE, or RMSE, all with the same theoretical argmin) with the desire to use MSE over SSE. $\endgroup$
    – Dave
    Aug 6, 2021 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.