Why do we usually choose to minimize the sum of square errors (SSE) when fitting a model?

Question

The question is very simple: why, when we try to fit a model to our data, linear or non-linear, do we usually try to minimize the sum of the squares of errors to obtain our estimator for the model parameter? Why not choose some other objective function to minimize? I understand that, for technical reasons, the quadratic function is nicer than some other functions, e.g., sum of absolute deviation. But this is still not a very convincing answer. Other than this technical reason, why in particular are people in favor of this 'Euclidean type' of distance function? Is there a specific meaning or interpretation for that?

The logic behind my thinking is the following:

When you have a dataset, you first set up your model by making a set of functional or distributional assumptions (say, some moment condition but not the entire distribution). In your model, there are some parameters (assume it is a parametric model), then you need to find a way to consistently estimate these parameters and hopefully, your estimator will have low variance and some other nice properties. Whether you minimize the SSE or LAD or some other objective function, I think they are just different methods to get a consistent estimator. Following this logic, I thought people use least square must be 1) it produces consistent estimator of the model 2) something else that I don't know.

In econometrics, we know that in linear regression model, if you assume the error terms have 0 mean conditioning on the predictors and homoscedasticity and errors are uncorrelated with each other, then minimizing the sum of square error will give you a CONSISTENT estimator of your model parameters and by the Gauss-Markov theorem, this estimator is BLUE. So this would suggest that if you choose to minimize some other objective function that is not the SSE, then there is no guarantee that you will get a consistent estimator of your model parameter. Is my understanding correct? If it is correct, then minimizing SSE rather than some other objective function can be justified by consistency, which is acceptable, in fact, better than saying the quadratic function is nicer.

In pratice, I actually saw many cases where people directly minimize the sum of square errors without first clearly specifying the complete model, e.g., the distributional assumptions (moment assumptions) on the error term. Then this seems to me that the user of this method just wants to see how close the data fit the 'model' (I use quotation mark since the model assumptions are probably incomplete) in terms of the square distance function.

A related question (also related to this website) is: why, when we try to compare different models using cross-validation, do we again use the SSE as the judgment criterion? i.e., choose the model that has the least SSE? Why not another criterion?

Answer 1

While your question is similar to a number of other questions on site, aspects of this question (such as your emphasis on consistency) make me think they're not sufficiently close to being duplicates.

Why not choose some other objective function to minimize?

Why not, indeed? If you objective is different from least squares, you should address your objective instead!

Nevertheless, least squares has a number of nice properties (not least, an intimate connection to estimating means, which many people want, and a simplicity which makes it an obvious first choice when teaching or trying to implement new ideas).

Further, in many cases people don't have a clear objective function, so there's an advantage to choosing what's readily available and widely understood.

That said, least squares also has some less-nice properties (sensitivity to outliers, for example) -- so sometimes people prefer a more robust criterion.

minimize the sum of square error will give you CONSISTENT estimator of your model parameters

Least squares is not a requirement for consistency. Consistency isn't a very high hurdle -- plenty of estimators will be consistent. Almost all estimators people use in practice are consistent.

and by Gauss-Markov theorem, this estimator is BLUE.

But in situations where all linear estimators are bad (as would be the case under extreme heavy-tails, say), there's not much advantage in the best one.

if you choose to minimize some other objective function that is not the SSE, then there is no guarantee that you will get consistent estimator of your model parameter. Is my understanding correct?

it's not hard to find consistent estimators, so no that's not an especially good justification of least squares

why when we try to compare different models using cross validation, we again, use the SSE as the judgment criterion? [...] Why not other criterion?

If your objective is better reflected by something else, why not indeed?

There is no lack of people using other objective functions than least squares. It comes up in M-estimation, in least-trimmed estimators, in quantile regression, and when people use LINEX loss functions, just to name a few.

was thinking that when you have a dataset, you first set up your model, i.e. make a set of functional or distributional assumptions. In your model, there are some parameters (assume it is a parametric model),

Presumably the parameters of the functional assumptions are what you're trying to estimate - in which case, the functional assumptions are what you do least squares (or whatever else) around; they don't determine the criterion, they're what the criterion is estimating.

On the other hand, if you have a distributional assumption, then you have a lot of information about a more suitable objective function -- presumably, for example, you'll want to get efficient estimates of your parameters -- which in large samples will tend to lead you toward MLE, (though possibly in some cases embedded in a robustified framework).

then you need to find a way to consistently estimate these parameters. Whether you minimize the SSE or LAD or some other objective function,

LAD is a quantile estimator. It's a consistent estimator of the parameter it should estimate in the conditions in which it should be expected to be, in the same way that least squares is. (If you look at what you show consistency for with least squares, there's corresponding results for many other common estimators. People rarely use inconsistent estimators, so if you see an estimator being widely discussed, unless they're talking about its inconsistency, it's almost certainly consistent.*)

* That said, consistency isn't necessarily an essential property. After all, for my sample, I have some particular sample size, not a sequence of sample sizes tending to infinity. What matters are the properties at the $n$ I have, not some infinitely larger $n$ that I don't have and will never see. But much more care is required when we have inconsistency - we may have a good estimator at $n$=20, but it may be terrible at $n$=2000; there's more effort required, in some sense, if we want to use consistent estimators.

If you use LAD to estimate the mean of an exponential, it won't be consistent for that (though a trivial scaling of its estimate would be) -- but by the same token if you use least squares to estimate the median of an exponential, it won't be consistent for that (and again, a trivial rescaling fixes that).

Answer 2

You asked a statistics question, and I hope that my control system engineer answer is a stab at it from enough of a different direction to be enlightening.

Here is a "canonical" information-flow form for control system engineering:

The "r" is for reference value. It is summed with an "F" transform of the output "y" to produce an error "e". This error is the input for a controller, transformed by the control transfer function "C" into a control input for the plant "P". It is meant to be general enough to apply to arbitrary plants. The "plant" could be a car engine for cruise control, or the angle of input of an inverse-pendulum.

Let's say you have a plant with a known transfer function with phenomenology suitable to the the following discussion, a current state, and a desired end state. (table 2.1 pp68) There are an infinite number of unique paths that the system, with different inputs, could traverse to get from the initial to final state. The textbook controls engineer "optimal approaches" include time optimal (shortest time/bang-bang), distance optimal (shortest path), force optimal (lowest maximum input magnitude), and energy optimal (minimum total energy input).

Just like there are an infinite number of paths, there are an infinite number of "optimals" - each of which selects one of those paths. If you pick one path and say it is best then you are implicitly picking a "measure of goodness" or "measure of optimality".

In my personal opinion, I think folks like L-2 norm (aka energy optimal, aka least squared error) because it is simple, easy to explain, easy to execute, has the property of doing more work against bigger errors than smaller ones, and leaves with zero bias. Consider h-infinity norms where the variance is minimized and bias is constrained but not zero. They can be quite useful, but they are more complex to describe, and more complex to code.

I think the L2-norm, aka the energy-minimizing optimal path, aka least squared error fit, is easy and in a lazy sense fits the heuristic that "bigger errors are more bad, and smaller errors are less bad". There are literally an infinite number of algorithmic ways to formulate this, but squared error is one of the most convenient. It requires only algebra, so more people can understand it. It works in the (popular) polynomial space. Energy-optimal is consistent with much of the physics that comprise our perceived world, so it "feels familiar". It is decently fast to compute and not too horrible on memory.

If I get more time I would like to put pictures, codes, or bibliographic references.

Answer 3

I think that, when fitting models, we usually choose to minimize the sum of squared errors ($SSE$) due to the fact that $SSE$ has a direct (negative) relation with $R^2$, a major goodness-of-fit (GoF) statistic for a model, as follows ($SST$ is sum of squares total):

$$ R^2 = 1 - \frac{SSE}{SST} $$

Omitting the discussion of why an adjusted $R^2$ is a better (unbiased) GoF statistic due to correction for sample size and number of coefficients (see this or this), it seems to me that this connection is important as $R^2$ family of statistics is the one that represents relative measures of the fit versus absolute measures, such as root mean squared error ($RMSE$).

Moreover, the fact that $R^2$ is equal to the percentage of the variance in the dependent variable that can be explained by all of the independent variables taken together, makes $R^2$ and, thus, indirectly, $SSE$, measures of explanatory power (or predictive power) of a model. In fact, for predictive models, some people recommend using a similar to $SSE$ statistic - predicted residual sum of squares ($PRESS$). For details, see this post and this post, which are relevant to your question in the end of the post.

Concluding and answering your main question, I think that we usually minimize $SSE$, because it is equivalent to maximizing explanatory or predictive power of a statistical model in question.

Answer 4

You might also look at minimizing the maximum error instead of least squares fitting. There is an ample literature on the subject. For a search word, try "Tchebechev" also spelled "Chebyshev" polynomials.

Answer 5

It looks people use squares because it allow to be within Linear Algebra realm and not touch other more complicated stuff like convex optimization which is more powerfull, but it lead to usin solvers without nice closed-form solutions.

Also idea from this math realm which has name convex optimization has not spread a lot.

"...Why do we care about square of items. To be honest because we can analyze it...If you say that it correspond to Energy and they buy it then move on quickly...." -- https://youtu.be/l1X4tOoIHYo?t=1416, EE263, L8, 23:36.

Also here Stephen P. Boyd describes in 2008 that people use hammer and adhoc: L20, 01:05:15 -- https://youtu.be/qoCa7kMLXNg?t=3916

Answer 6

On a side note:

When factoring in uncertanty over the values of our target variable t, we can express the probability distribution of t as $$p(t|x,w,\beta) = \mathbb{N}(t|y(x,\textbf{w}),\beta^{-1})$$ assuming t follows a Gaussian conitioned on the polyomial y. Using training data $\{\textbf{x}, \textbf{t}\}$ the likelihood for the model parameters $\textbf{w}$ is given by $$ p(\textbf{t}|\textbf{x}, \textbf{w}, \beta) = \prod_{n=1}^ {N}\mathbb{N}(t_n|y(x_n, \textbf{w}),\beta^{-1}).$$ Maximizing the log likelihood of the form $$-\frac{\beta}{2}\sum_{n=1}^{N}\{y(x_n, \textbf{w})-t_n\}^2 + \frac{N}{2}ln\beta-\frac{N}{2}ln(2\pi)$$ is the same as minimizing the negative log likelihood. We cab drop the second and the third term since they're constant with regards to $\textbf{w}$. Also the scaling factor $\beta$ in the first term can be dropped, since a constant factor does not change the location of the maximum/minimum, leaving us with $$-\frac{1}{2}\sum_{n=1}^{N}\{y(x_n, \textbf{w})-t_n\}^2.$$ Thus the SSE has arisen as a consequence of maximizing likelihood under the assumption of a Gaussian noise distribution.