After a lot of discussions and having the luck to talk with some phds, I want to share this answer that I really like.
Let's say we have a model that we trained.
How can we say if that is a good or bad model?
Well, to answer this question we need something to validate our model.
For that we are using something that we call a metric. In math we know many metrics, which are able to measure stuff.
Now there are different ways to measure stuff. In our case we want to measure the errors to see how good our trained model is.
The best would be if our metric can give us something that is easy to reason about and interpret.
The metric that we pick, should be something that has some meaning in human terms, so with other words it should be easy to reason about and to compare with other values and to interpret.
Because of these requirement, one idea is to use a metric, that square sums all errors up and gives us the mean of this sum.
The root of this mean is a representative that tells us that if we do not have outliers, that the values of our model, should be close to this mean value.
To understand that let's say that our trained model gets 5 input values and outputs [3, 3, 4, 4, 2].
Is our model good? Are these values a good prediction?
Well to answer this question, we use the metric that we used to train/optimize hour model.
So we use the mean of squared errors (MSE) and give it the same 5 values our model got.
Let's say we got 16. Now we calculate the square root and we get 4.
This 4 is a representative value that is telling us, that the best approximation for hour predictions should be close to 4.
This shows how we used the MSE to compare/evaluate/test our model.
When we train a regression model, we always can evaluate and test it with a metric, with other words we can see how good the model is.
Many ML frameworks use the same metric for training and test, because we can reuse the same implementation for training and testing/evaluation.
Looking at it in a mathematical point of view, the squared sum of errors (SSE) and the MSE have the same same minimization value.
If we take our cost function, or any function in general, that we want to optimize, which means to find a minimum or maximum, we can multiply any constant to it, the solutions will never change.
We remember what we learned in our calculus classes. We take the derivative of the function, set it to zero and then solve for the parameters.
Let f'(x) = x + 1. Now to find the extrem point, we have to set f'(x) to zero, so we get x + 1 = 0.
x + 1 = 0 has the same solution like a(x + 1) = a 0, so it does not matter what factor we use with our function we want to optimize.
This means min(SSE) = min(MSE), because the only difference between the SSE and the MSE is a factor (like the a in the line above) that does not influence the solution of the minimization.
This shows that the SSE can also be used to train our model, but it's bad to evaluate the model, because the meaning is hidden and hard to interpret, which is why we use the MSE.
So the reason why the literature prefers the MSE for training over the SSE is just a historic convention.
The convention says to use the metric for training, that you want to use to test your model with. Which is a nice and consistent convention.