Maybe this question is simple, but I really need some help. When we use the Maximum Likelihood Estimation(MLE) to estimate the parameters, why we always put the log() before the joint density? To use the sum in place of product? But why? The wikipedia said it will be convenient. Why? Thank you.
$\begingroup$
$\endgroup$
3
-
6$\begingroup$ We don't always do it. But besides the 'turn products into sums', there's also the fact that the numbers become far more tractable (computing pure likelihoods on typical to large sample sizes can often suffer from underflow problems), and finally there's the fact that a very large number of densities and probability functions involve exponentiation or powers, so even the individual terms may become nicer to work with. In fact, there are typically so many reasons to do it, you would usually only avoid it with good reason. $\endgroup$– Glen_bSep 25, 2013 at 2:07
-
4$\begingroup$ If your data consists of $n$ independent samples, then the likelihood function is the product of the likelihood functions of the individual samples. To maximize the likelihood, differentiate the likelihood etc, which results in the sum of $n$ terms and is sometimes a mess to work with. If we take the logarithm and then try to maximize the logarithm of the likelihood (which yields the same maximum) we have the derivative is the sum of $n$ terms each of which depends on a separate sample value: much easier to work with than a sum of $n$ terms that all depend on all the $n$ sample values. $\endgroup$– Dilip SarwateSep 25, 2013 at 2:09
-
1$\begingroup$ Another viewpoint: if $f(x)>0$, for every $x$, then $(\log f(x))'=f'(x)/f(x)=0$ if and only if $f'(x)=0$. Hence, maximization of the log-likelihood gives you the same answer (the mle), and, as already pointed, generally makes things analytically simpler. $\endgroup$– ZenSep 25, 2013 at 3:09
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Apart from the reasons mentioned in the comments to your question, there is another important one: in applying maximum likelihood estimation, we essentially solve a maximization problem with respect to the unknown coefficients. Recall that finding the global maximum of a function is not a simple matter, in case where we have many unknowns, and when the objective function lacks (or is unknown whether it possesses) certain general properties, like concavity (in the case of maximization), especially when the maximization will be done through an iterative procedure (as will be the case for most likelihood functions). Moreover, concavity of the objective function is an important condition in proving consistency of the ML estimator when the parameter space is not compact (for example when you estimate variances, $\sigma^2$, the parameter space is not compact but open from below, since by conception $\sigma^2 >0$.
So we would want our objective function to be concave with respect to the parameters, to guarantee a global maximum. In linear models, if we have concavity in the variable, we obtain concavity in the parameters. Now there are many widely used distributions, whose density functions are not concave, but their natural logarithms are (we call such functions "log-concave"). The Normal density is the most prominent example: the function $$f_X(x) =\frac {1}{\sigma\sqrt{2\pi}}e^{-\frac 12 (\frac{x-\mu}{\sigma})^2} $$ is neither convex nor concave in $x$ (it has a middle concave part and is convex in the tails). But the function $$\ln f_X(x) =\ln \left(\frac {1}{\sigma\sqrt{2\pi}}\right) -\frac 12 \left(\frac{x-\mu}{\sigma}\right)^2 $$
is globally concave in $x$. (Then by using the invariance property of the ML estimator, we can show by a suitable one-to-one transformation of the unknown parameter vector, that the function is concave in the re-parametrized vector).
But in general, the basic point is that taking logs produces concavity of the objective function, which is a very desirable property.
answered Sep 25, 2013 at 2:51
-
1$\begingroup$ +1 The point about convexity of $-\log(L)$ is good, although perhaps overspecialized (it does not apply to many likelihoods). But I believe the answer goes much deeper than that: as statisticians, we are (or at least should be) at least as interested in the possible errors in our estimates as we are in the estimates themselves. Those errors are computed from relatively simple properties of $-\log(L)$ and not from properties of $L$ itself. $\endgroup$– whuber ♦Sep 25, 2013 at 15:09
$\begingroup$
$\endgroup$
1
In addition to the mathematical reason that Alecos wrote, let me give you a computational reason. Remember that the likelihood function is nothing but the joint density of random variables (expressed as a function of the parameters), i.e. $$ Pr(\mathbf{x}) = Pr(x_{1})\cdot Pr(x_{2})\cdot\ldots\cdot Pr(x_{n}) = \prod_{i}^{n} Pr(x_{i}) $$ for i.i.d. data. The probability density $0 \leq Pr(x_{i}) \leq 1$ for all $i$, so this number $Pr(\mathbf{x})$ becomes very small quickly as $n$ increases. Suppose all $Pr(x_{i}) = 0.5$ and $n=1000$, then $$ \prod_{i}^{n} Pr(x_{i}) = 0.5^{1000} = 9.33 \cdot 10^{-302} $$ For only slightly larger datasets, or slightly smaller $Pr(x_{i})$, we are outside the representable range for software packages. For instance, the smallest representable number in R is $2.225074\cdot10^{-308}$. On the flipside, we have $$ \log(Pr(\mathbf{x})) = \sum_{i}^{n} \log \left( Pr(x_{i}) \right) = 1000\cdot \log(0.5) = -693.1472 $$ and even for $n=1000000$ we only have $\log(Pr(\mathbf{x})) = 1000000\cdot \log(0.5) = -693147.2$.
-
$\begingroup$ Similarly, the gradients are much easier for a large summation compared with a large product. $\endgroup$– Cliff ABMar 10, 2019 at 23:39