Why is the "de-facto" in statistics to minimize the sum of squared errors cost function instead of maximizing some reward function like the likelihood function?
$\begingroup$
$\endgroup$
1
-
6$\begingroup$ If you define loss = -reward, aren't they the same? $\endgroup$– Glen_bCommented Apr 23, 2014 at 1:56
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
It's important to understand why we typically work with the log of the likelihood rather than the likelihood function itself. Since the logarithmic transformation is monotone, maximizing the log likelihood is equivalent to maximizing the likelihood. For a linear model with a multivariate normal likelihood function, the log of the likelihood function is minus the sum of squares. Maximizing minus the sum of squares is equivalent to minimizing the sum of squares. Thus in this special case maximum likelihood estimation and least squares are mathematically equivalent.
The are two important reasons why maximizing the log likelihood is often a better choice than maximizing the likelihood itself. First, the log likelihood function typically has a much smaller range of values than the likelihood. It's easy to get into situations where the likelihood function underflows to 0 in floating point arithmetic. A second important point is that in many cases the log likelihood function is a concave function while the likelihood itself is not concave. Maximizing a concave function is much more computationally tractable than trying to directly maximize a function that is only log-concave.
answered Apr 23, 2014 at 1:50
$\begingroup$
$\endgroup$
1
You don't always have the likelihood function, while likelihood maximization assumes that you know the distribution of errors. Minimizing the squared errors you don't need the distribution, it's convenient. It's convenient to be able to get an optimal solution in some respect than not be able to any solution at all.
This is the answer to why least squares over maximum likelihood. If you're interested why squared errors and not other cost function, then it's a different story. Least squares are very convenient mathematically, in comparison to the sum of absolute errors, another popular choice of cost functions, due to its smooth first derivative. Also, it produces the forecasts which are expectations of variables, unlike the absolute error costs which lead to medians. Why means are better than medians? Well... that's a long story :)
-
1$\begingroup$ you should define "convenient". I find that as a very unsatisfactory answer because its vague. Do you mean its convenient because the squared loss function is convex/concave and thus its easier to find a global max? or what do you mean? Or do you mean its convenient because it otherwise transforms a NP-complete problem to something manageable? Or did you mean? $\endgroup$ Commented Apr 23, 2014 at 1:19
$\begingroup$
$\endgroup$
To add to what Aksakal said, minimizing the squared error isn't the only optimization criterion for selecting a model, and in some cases, like OLS regression, they give you the same parameter estimates.
