1
$\begingroup$

I understand that the MLE considers the joint probability of the observed data. So consider, for example, we are trying to estimate the mean $\mu$ from a random sample distributed iid according to the normal density function $\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$ given $X, X=\{x_1,x_2,\cdots,x_n\}$ with $\mu, \sigma^2$ unknown. We then consider the joint probability as $P\{X\} = \prod_i^n P\{x_i\} = \prod_i^n \frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x_i-\mu)^2}{2\sigma^2}}$ by independence. The we use the log, set the equation $=0$ and solve. But I have always wondered why the integral is not at all present in the formulation. Perhaps I have written it wrong but what am I not understanding.

$\endgroup$
1
  • 2
    $\begingroup$ The joint probability of observing $(X_1, X_2,\ldots, X_n) = (x_1,x_2,\ldots, x_n)$ is $0$. Instead, the value of the likelihood is defined to be the joint density of $(X_1, X_2,\ldots, X_n)$ evaluated at $(x_1,x_2,\ldots, x_n)$, and considered as a function of the parameter $\mu$. That is, $$L(\mu; x_1,x_2,\ldots,x_n) = f(x_1,x_2,\ldots, x_n; \mu).$$ And for heaven's sake don't murmur the usual shibboleths of "take logs, set derivative$=0$" etc. $L(\mu)$ is in the form of a normal density function (times a constant) and has a maximum at the mean of this normal density function. $\endgroup$ – Dilip Sarwate Sep 13 '15 at 22:05
1
$\begingroup$

First let me touch up a few things in your description of MLE. On the left hand side what we have is $P(X | \mu)$. This is important because we are thinking about this as a function of $\mu$.

Then we take a log (often) because it makes the expression easier to deal with because it breaks the product into a sum and brings down powers of exponential a where parameters (such as $\mu$ often reside). The reason we can do this is that log is monotone increasing so $log(f(x))$ is maximized at the same $x$ as $f(x)$.

We do not set this equal to zero. We take the derivative and set it equal to zero as we do in any unconstrained smooth optimization problem to find critical points.

The reason that I go into this detail is that if you understand why we do what we do, you may understand why we don't do what we don't do. Integration typically means one of two things. Finding the expected value of a function of a random variable (don't need it) or marginalizing out a variable from a joint distribution (don't need it).

$\endgroup$
0
$\begingroup$

The product is indexed over the countable list of your random sample, as indicated by the superscript $n$ over the Pi operator.

You are confusing the case in probability theory where a random variable following a known distribution can summarize the probability of a range of events. When those random variables have a continuous distribution, you integrate probability density over the sample space. When those random variables have a discrete distribution, you sum probabilities over the sample space.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.