2
$\begingroup$

MLE for independent data samples $D = (x_1,y_1), (x_2,y_2), (x_3,y_3) ... (x_N, y_N)$ can be formulated as

$$ L(D) = \prod_{i=1}^N p_i(x_i, y_i) $$

And the log likelihood being:

$$ \log(L(D)) = \sum_{i=1}^N \log(p_i(x_i, y_i)) $$

However, I have seen in numerous literature the log likelihood being written as

$$ \log(L(D)) = \frac{1}{N} \times \sum_{i=1}^N \log(p_i(x_i, y_i)) $$

I have some questions:

  1. Why is the log likelihood divided by sample size? This is no effect on parameter estimation in my opinion.
  2. Is it assumed that the data generating process is I.I.D. for this division to make sense?
Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ Dividing by $N$ is useful when considering the limit as $N$ goes to infinity, not for estimating the (implicit) parameter. This does not require the data to be iid (but convergence may prove harder to establish). $\endgroup$
    – Xi'an
    Commented Apr 28 at 10:47
  • $\begingroup$ @Xi'an thank you! With regards to question 2. is there any connection with cross-entropy as mentioned here ? $\endgroup$
    – spie227
    Commented Apr 28 at 11:11
  • 1
    $\begingroup$ Re #2: the initial formulation of $L$ is justified only by the assumption of independence (and is equivalent to it). It would be useful to have a clear, accurate quotation from the "literature," because obviously division by $N$ changes the log likelihood, making it incorrect to refer to it as one. $\endgroup$
    – whuber ♦
    Commented Apr 28 at 18:18
  • 1
    $\begingroup$ What @Xi'an mentioned in the first comment however makes more sense to me as to why division by N is considered especially as it does not make any assumptions about the data generating process. $\endgroup$
    – spie227
    Commented Apr 29 at 15:50
  • 1
    $\begingroup$ I think you read more into that quotation than it says. At the very least, it clearly explains why it is dividing by $N$ and, in this context, makes no implication that the result is any kind of likelihood. $\endgroup$
    – whuber ♦
    Commented Apr 29 at 17:18

1 Answer 1

Reset to default
3
$\begingroup$

Here is a picture I use in my mathematical statistics lecture

enter image description here

that represents $$L(\theta|x_1,\ldots,x_n)^{1/n}=\exp\left\{\frac{1}{n}\sum_{i=1}^n p(x_i|\theta)\right\}$$ when the $x_i$'s are iid Poisson $\mathcal P(\theta_0)$ with $\theta_0=2$ and $n$ goes from $5$ to $50$. It is intended to show the progressive stabilisation of this function with $n$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.