I was reading about Maximum likelihood estimation from various sources on the internet and I noticed that MLE makes an assumption about the data known as IID but I didn't completely understand why is it necessary to make this assumption? Are there any other assumptions that MLE makes?
$\begingroup$
$\endgroup$
8
-
4$\begingroup$ It isn't necessary. MLE maximizes a wide array of likelihoods. Your sources are assuming iid because it is a reasonable modeling assumption in a particular setting. If your model assumes the data are iid, it is easy to write down the likelihood and then use MLE to estimate the unknown parameters. $\endgroup$– TaylorCommented Jan 8, 2019 at 23:38
-
3$\begingroup$ ... it's reasonable and it's extremely convenient, because it implies that the overall likelihood of the data set is the product of the likelihoods of the individual observations (independent), which all have the same form (identically distributed) $\to$ log-likelihood is the sum of individual LLs ... $\endgroup$– Ben BolkerCommented Jan 8, 2019 at 23:40
-
1$\begingroup$ @Taylor I didn't quite understand 'wide array of likelihoods'. Can you name a few? $\endgroup$– Akash DubeyCommented Jan 8, 2019 at 23:40
-
3$\begingroup$ Consider a time series model like an AR(1) process. Then the order of the data matter, and your likelihood is a product of conditional densities, not marginal densities. If you have three data points, you could write the likelihood as $p(x_3 \mid x_2) p(x_2 \mid x_1) p(x_1)$. MLE would have no problem with this, but we are not assuming the data are iid. $\endgroup$– TaylorCommented Jan 8, 2019 at 23:44
-
2$\begingroup$ Possible duplicate of What does it mean by independently and identically distributed random variables? $\endgroup$– TaylorCommented Jan 8, 2019 at 23:45
|
Show 3 more comments
1 Answer
$\begingroup$
$\endgroup$
6
Assuming independence is not necessary for maximum likelihood estimation (ML) (or other likelihood based methods). But if independence is a reasonable assumption, then it makes ML easy to implement, since the log likelihood is simply the sum of the individual log likelihoods. But there are lots of examples where ML is used without Independence: time series with ARMA (or ARIMA) models, spatial models including spatial dependence, mixed models where there are correlation between observations within some groups, others.
When is it reasonable to assume independence? is discussed already multiple times on this site, see for example Are "random sample" and "iid random variable" synonyms? or Independence of events in real-life data
answered Jan 9, 2019 at 23:05
-
$\begingroup$ How does it makes ML easy to implement. Can you edit your answer and explain how it makes the maths easy? $\endgroup$ Commented Jan 9, 2019 at 23:41
-
-
$\begingroup$ Can you share any link for now? Or any reference or book? $\endgroup$ Commented Jan 9, 2019 at 23:43
-
$\begingroup$ It’s worth pointing out that the theory for MLE is easiest to develop for iid data. So it makes sense for textbooks to focus on this case. $\endgroup$– guyCommented Jan 10, 2019 at 0:05
-
1$\begingroup$ @guy I tired looking for the same on the internet but couldn't get any. It would be great if you can share any references for the same. $\endgroup$ Commented Jan 10, 2019 at 5:49