The question is very simple: why, when we try to fit a model to our data, linear or non-linear, do we usually try to minimize the sum of the squares of errors to obtain our estimator for the model parameter? Why not choose some other objective function to minimize? I understand that, for technical reasons, the quadratic function is nicer than some other functions, e.g., sum of absolute deviation. But this is still not a very convincing answer. Other than this technical reason, why in particular are people in favor of this 'Euclidean type' of distance function? Is there a specific meaning or interpretation for that?
The logic behind my thinking is the following:
When you have a dataset, you first set up your model by making a set of functional or distributional assumptions (say, some moment condition but not the entire distribution). In your model, there are some parameters (assume it is a parametric model), then you need to find a way to consistently estimate these parameters and hopefully, your estimator will have low variance and some other nice properties. Whether you minimize the SSE or LAD or some other objective function, I think they are just different methods to get a consistent estimator. Following this logic, I thought people use least square must be 1) it produces consistent estimator of the model 2) something else that I don't know.
In econometrics, we know that in linear regression model, if you assume the error terms have 0 mean conditioning on the predictors and homoscedasticity and errors are uncorrelated with each other, then minimizing the sum of square error will give you a CONSISTENT estimator of your model parameters and by the Gauss-Markov theorem, this estimator is BLUE. So this would suggest that if you choose to minimize some other objective function that is not the SSE, then there is no guarantee that you will get a consistent estimator of your model parameter. Is my understanding correct? If it is correct, then minimizing SSE rather than some other objective function can be justified by consistency, which is acceptable, in fact, better than saying the quadratic function is nicer.
In pratice, I actually saw many cases where people directly minimize the sum of square errors without first clearly specifying the complete model, e.g., the distributional assumptions (moment assumptions) on the error term. Then this seems to me that the user of this method just wants to see how close the data fit the 'model' (I use quotation mark since the model assumptions are probably incomplete) in terms of the square distance function.
A related question (also related to this website) is: why, when we try to compare different models using cross-validation, do we again use the SSE as the judgment criterion? i.e., choose the model that has the least SSE? Why not another criterion?