I am learning Maximum Likelihood Estimation.

Per this post, the log of the PDF for a normal distribution looks like this:

$$ \log{\left(f\left(x_i;\,\mu,\sigma^2\right)\right)} = - \frac{n}{2} \log{\left(2 \pi\right)} - \frac{n}{2} \log{\left(\sigma^2\right)} - \frac{1}{2 \sigma^2} \sum{{\left(x_i - \mu\right)}^2} \tag{1} $$

According to any Probability Theory textbook, the formula of the PDF for a normal distribution: $$ \frac {1}{\sigma \sqrt {2\pi}} e^{-\frac {(x - \mu)^2}{2\sigma ^2}} \rlap{\qquad \text{where}~-\infty <x<\infty} \tag{2} $$

Taking the log of Expression 2 produces

\begin{align} \ln\left(\frac{1}{\sigma \sqrt {2\pi}} e^{-\frac{\left(x - \mu\right)^2}{2\sigma ^2}}\right) &= \ln\left(\frac {1}{\sigma \sqrt {2\pi}}\right)+\ln{\left(e^{-\frac {(x - \mu)^2}{2\sigma ^2}}\right)} \tag{3} \\[5px] &=-\ln\left(\sigma\right)-\frac{1}{2} \ln\left(2\pi\right) - \frac{\left(x - \mu\right)^2}{2\sigma ^2} \tag{4} \end{align}

which is very different from Equation 1.

Is Equation 1 right? What am I missing?

  • 7
    $\begingroup$ Your first equation is the joint log-pdf of a sample of n iid normal random variables (AKA the log-likelihood of that sample). The second equation is the the log-pdf of a single normal random variable $\endgroup$ – Artem Mavrin Apr 20 at 22:26
  • 2
    $\begingroup$ @ArtemMavrin, I think your comment would be a perfectly good answer if you expanded on just a bit to make it slightly more clear. $\endgroup$ – StatsStudent Apr 20 at 22:47

For a single observed value $x$ you have log-likelihood:

$$\ell_x(\mu,\sigma^2) = - \ln \sigma - \frac{1}{2} \ln (2 \pi) - \frac{1}{2} \Big( \frac{x-\mu}{\sigma} \Big)^2.$$

For a sample of observed values $\mathbf{x} = (x_1,...,x_n)$ you then have:

$$\ell_\mathbf{x}(\mu,\sigma^2) = \sum_{i=1}^n \ell_{x_i}(\mu,\sigma^2) = - n \ln \sigma - \frac{n}{2} \ln (2 \pi) - \frac{1}{2 \sigma^2} \sum_{i=1}^n (x_i-\mu)^2.$$

(Note that in both cases you can remove the constant term from the log-likelihood, but then it is not the logarithm of the sampling density.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.