2
$\begingroup$

I was reading about Maximum likelihood estimation from various sources on the internet and I noticed that MLE makes an assumption about the data known as IID but I didn't completely understand why is it necessary to make this assumption? Are there any other assumptions that MLE makes?

$\endgroup$
  • 2
    $\begingroup$ It isn't necessary. MLE maximizes a wide array of likelihoods. Your sources are assuming iid because it is a reasonable modeling assumption in a particular setting. If your model assumes the data are iid, it is easy to write down the likelihood and then use MLE to estimate the unknown parameters. $\endgroup$ – Taylor Jan 8 at 23:38
  • 1
    $\begingroup$ ... it's reasonable and it's extremely convenient, because it implies that the overall likelihood of the data set is the product of the likelihoods of the individual observations (independent), which all have the same form (identically distributed) $\to$ log-likelihood is the sum of individual LLs ... $\endgroup$ – Ben Bolker Jan 8 at 23:40
  • 1
    $\begingroup$ @Taylor I didn't quite understand 'wide array of likelihoods'. Can you name a few? $\endgroup$ – Akash Dubey Jan 8 at 23:40
  • 1
    $\begingroup$ Consider a time series model like an AR(1) process. Then the order of the data matter, and your likelihood is a product of conditional densities, not marginal densities. If you have three data points, you could write the likelihood as $p(x_3 \mid x_2) p(x_2 \mid x_1) p(x_1)$. MLE would have no problem with this, but we are not assuming the data are iid. $\endgroup$ – Taylor Jan 8 at 23:44
  • 2
    $\begingroup$ Possible duplicate of What does it mean by independently and identically distributed random variables? $\endgroup$ – Taylor Jan 8 at 23:45
2
$\begingroup$

Assuming independence is not necessary for maximum likelihood estimation (ML) (or other likelihood based methods). But if independence is a reasonable assumption, then it makes ML easy to implement, since the log likelihood is simply the sum of the individual log likelihoods. But there are lots of examples where ML is used without Independence: time series with ARMA (or ARIMA) models, spatial models including spatial dependence, mixed models where there are correlation between observations within some groups, others.

When is it reasonable to assume independence? is discussed already multiple times on this site, see for example Are "random sample" and "iid random variable" synonyms? or Independence of events in real-life data

$\endgroup$
  • $\begingroup$ How does it makes ML easy to implement. Can you edit your answer and explain how it makes the maths easy? $\endgroup$ – Akash Dubey Jan 9 at 23:41
  • $\begingroup$ Will try tomorrow! $\endgroup$ – kjetil b halvorsen Jan 9 at 23:42
  • $\begingroup$ Can you share any link for now? Or any reference or book? $\endgroup$ – Akash Dubey Jan 9 at 23:43
  • $\begingroup$ It’s worth pointing out that the theory for MLE is easiest to develop for iid data. So it makes sense for textbooks to focus on this case. $\endgroup$ – guy Jan 10 at 0:05
  • $\begingroup$ @guy I tired looking for the same on the internet but couldn't get any. It would be great if you can share any references for the same. $\endgroup$ – Akash Dubey Jan 10 at 5:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.